00000800:00000200:2.1F:1734015829.684531:0:1979:0:(socklnd.c:225:ksocknal_find_peer_locked()) got peer_ni [ffff88012b2c8780] -> 12345-192.168.201.116@tcp (2) 00000800:00000200:0.1F:1734015829.689683:0:1968:0:(socklnd.c:409:ksocknal_incr_conn_count()) Add conn type 1, ksnr_connected 2 ksnr_max_conns 1 00000800:00000200:0.1:1734015829.689695:0:1968:0:(socklnd_cb.c:748:ksocknal_queue_tx_locked()) Sending to 12345-192.168.201.116@tcp ip 192.168.201.116:988 00000800:00000200:0.1:1734015829.689699:0:1968:0:(socklnd_cb.c:767:ksocknal_queue_tx_locked()) Packet ffff8800b631a200 type 2, nob 96 niov 1 nkiov 0 00000800:00000200:2.1:1734015829.692405:0:1968:0:(socklnd.c:409:ksocknal_incr_conn_count()) Add conn type 2, ksnr_connected 6 ksnr_max_conns 1 00000800:00000200:0.1:1734015829.700726:0:1968:0:(socklnd.c:409:ksocknal_incr_conn_count()) Add conn type 3, ksnr_connected e ksnr_max_conns 1 00000020:00010000:3.0:1734016133.607222:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1915, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.607224:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 40 00000020:00010000:3.0:1734016133.607226:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016133.607227:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 40 00000100:00100000:3.0:1734016133.607240:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335345536:12345-192.168.201.116@tcp:106: Request processed in 37us (1051us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.607245:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12122 00000100:00100000:3.0:1734016133.613624:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016133.622811:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335345792 00000100:00100000:3.0:1734016133.622819:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335345792 00000100:00100000:3.0:1734016133.622859:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12124 00000100:00100000:3.0:1734016133.622867:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335345792:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016133.622875:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [208896->221183] (req 208896->221183) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.622883:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [208896->221183] (req 208896->221183) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.622890:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.622898:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.622905:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.622913:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.622932:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c1d40) 00000100:00100000:3.0:1734016133.622965:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335345792:12345-192.168.201.116@tcp:105: Request processed in 100us (365us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.622971:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12124 00000100:00100000:3.0:1734016133.631333:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016133.631347:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335345920 00000100:00100000:3.0:1734016133.631354:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335345920 00000100:00100000:3.0:1734016133.631385:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12125 00000100:00100000:3.0:1734016133.631392:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dcc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335345920:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.631405:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.631414:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.631433:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.631435:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.631438:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016133.631439:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.631538:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dcc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335345920:12345-192.168.201.116@tcp:106: Request processed in 146us (352us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.631545:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12125 00010000:00010000:3.0:1734016133.634917:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.634936:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.634946:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.634951:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.634955:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.634967:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3643390, setting rss=3643390; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.634976:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.634982:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.634989:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.634994:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.635000:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.635013:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.635018:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.635022:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e6418 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.635049:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333845120:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.637926:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335346048 00000100:00100000:3.0:1734016133.637935:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335346048 00000100:00100000:3.0:1734016133.638221:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12126 00000100:00100000:3.0:1734016133.638228:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335346048:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.638241:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.638250:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.638280:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.638282:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.638285:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016133.638286:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.638404:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335346048:12345-192.168.201.116@tcp:106: Request processed in 174us (1372us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.638411:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12126 00010000:00010000:3.0:1734016133.641058:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.641073:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.641085:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.641090:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.641096:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.641109:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3643390, setting rss=3643390; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.641120:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.641125:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.641132:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.641137:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.641141:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.641150:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.641153:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.641158:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e641f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.641705:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333845248:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.645982:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335346176 00000100:00100000:3.0:1734016133.645991:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335346176 00000100:00100000:3.0:1734016133.646026:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12127 00000100:00100000:3.0:1734016133.646032:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335346176:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016133.646046:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e63cb lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271165ad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.651962:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016133.651974:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335346176:12345-192.168.201.116@tcp:104: Request processed in 5939us (6287us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.651982:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12127 00000100:00100000:3.0:1734016133.660971:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335346304 00000100:00100000:3.0:1734016133.660981:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335346304 00000100:00100000:3.0:1734016133.661020:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12128 00000100:00100000:3.0:1734016133.661026:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335346304:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016133.661040:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1843200->1851391] (req 1843200->1851391) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.661055:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1843200->1851391] (req 1843200->1851391) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.661062:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.661071:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.661079:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.661088:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.661110:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366d00) 00000100:00100000:3.0:1734016133.661163:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335346304:12345-192.168.201.116@tcp:105: Request processed in 134us (1366us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.661167:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12128 00000100:00100000:3.0:1734016133.668347:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333845632:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016133.673818:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335346432 00000100:00100000:3.0:1734016133.673827:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335346432 00000100:00100000:3.0:1734016133.673852:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335346560 00000100:00100000:3.0:1734016133.673854:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335346560 00000100:00100000:3.0:1734016133.673865:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12129 00000100:00100000:3.0:1734016133.673870:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335346432:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.673884:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.673893:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.673916:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.673918:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.673921:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016133.673922:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.673964:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335346432:12345-192.168.201.116@tcp:106: Request processed in 93us (816us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.673970:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12129 00000100:00100000:3.0:1734016133.673982:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12130 00000100:00100000:3.0:1734016133.673985:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335346560:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.673990:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.673995:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.674004:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.674005:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.674007:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016133.674008:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.674021:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335346560:12345-192.168.201.116@tcp:106: Request processed in 35us (502us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.674026:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12130 00010000:00010000:3.0:1734016133.676529:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676550:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676561:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676567:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676575:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.676588:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.676594:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676600:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676607:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676615:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.676621:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676631:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.676635:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676641:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e642d lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.676675:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333845760:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016133.676690:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676696:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676702:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676708:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.676713:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.676720:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3643390, setting rss=3643390; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.677214:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.677221:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.677227:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.677231:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.677233:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.677239:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.677241:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.677246:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6434 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.677271:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333845888:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.677920:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a89a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333846016:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.683733:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016133.683748:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335346688 00000100:00100000:3.0:1734016133.683754:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335346688 00000100:00100000:3.0:1734016133.683780:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335346816 00000100:00100000:3.0:1734016133.683782:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335346816 00000100:00100000:3.0:1734016133.683792:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12131 00000100:00100000:3.0:1734016133.683797:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335346688:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.683808:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.683816:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.683836:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.683838:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.683841:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016133.683842:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.683880:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335346688:12345-192.168.201.116@tcp:106: Request processed in 82us (497us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.683885:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12131 00000100:00100000:3.0:1734016133.683896:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12132 00000100:00100000:3.0:1734016133.683900:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335346816:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.683905:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.683912:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.683922:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.683923:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.683926:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016133.683927:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.683942:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335346816:12345-192.168.201.116@tcp:106: Request processed in 41us (420us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.683946:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12132 00010000:00010000:3.0:1734016133.687590:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.687597:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.687603:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.687608:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.687612:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.687618:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3643390, setting rss=3643390; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.687625:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.687629:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.687634:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.687642:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.687645:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.687650:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.687653:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.687659:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6442 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.687673:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333846144:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.691020:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335346944 00000100:00100000:3.0:1734016133.691028:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335346944 00000100:00100000:3.0:1734016133.691137:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12133 00000100:00100000:3.0:1734016133.691144:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335346944:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016133.691157:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6426 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116616 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.691746:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016133.691754:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335346944:12345-192.168.201.116@tcp:104: Request processed in 609us (993us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.691760:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12133 00000100:00100000:3.0:1734016133.695913:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335347072 00000100:00100000:3.0:1734016133.695920:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335347072 00000100:00100000:3.0:1734016133.695952:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12134 00000100:00100000:3.0:1734016133.695958:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335347072:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016133.695965:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1810432->1818623] (req 1810432->1818623) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.695973:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1810432->1818623] (req 1810432->1818623) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.695979:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.695986:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.695992:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.695999:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.696014:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa364900) 00000100:00100000:3.0:1734016133.696049:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335347072:12345-192.168.201.116@tcp:105: Request processed in 91us (432us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.696055:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12134 00000008:00010000:3.0:1734016133.705868:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7780 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.705935:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7780 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.705958:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333846912:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016133.709683:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335347200 00000100:00100000:3.0:1734016133.709692:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335347200 00000100:00100000:3.0:1734016133.709732:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12135 00000100:00100000:3.0:1734016133.709738:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335347200:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.709753:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.709762:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.709784:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.709787:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.709790:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016133.709791:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.709836:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335347200:12345-192.168.201.116@tcp:106: Request processed in 96us (447us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.709843:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12135 00010000:00010000:3.0:1734016133.713435:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.713453:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.713588:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.713595:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.713601:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.713615:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.713655:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.713661:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.713668:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.713675:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.713679:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.713690:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.713695:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.713705:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6450 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.714151:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333847040:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.716752:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335347328 00000100:00100000:3.0:1734016133.716760:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335347328 00000100:00100000:3.0:1734016133.716795:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12136 00000100:00100000:3.0:1734016133.716800:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335347328:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.716812:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.716821:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.716843:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.716845:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.716848:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016133.716849:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.716886:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335347328:12345-192.168.201.116@tcp:106: Request processed in 86us (245us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.716891:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12136 00010000:00010000:3.0:1734016133.721028:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.721046:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.721065:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.721071:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.721077:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.721092:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.721196:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.721202:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.721208:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.721212:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.721217:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.721228:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.721232:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.721241:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6457 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.722208:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333847168:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.725737:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335347456 00000100:00100000:3.0:1734016133.725746:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335347456 00000100:00100000:3.0:1734016133.725779:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12137 00000100:00100000:3.0:1734016133.725785:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335347456:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.725798:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.725808:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.725831:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.725833:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.725835:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016133.725837:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.725876:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335347456:12345-192.168.201.116@tcp:106: Request processed in 90us (307us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.725882:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12137 00010000:00010000:3.0:1734016133.728745:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.728762:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.728773:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.728778:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.728783:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.728794:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.729028:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.729034:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.729040:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.729045:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.729049:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.729058:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.729062:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.729072:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e645e lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.729207:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333847296:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.733021:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335347584 00000100:00100000:3.0:1734016133.733030:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335347584 00000100:00100000:3.0:1734016133.733066:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12138 00000100:00100000:3.0:1734016133.733073:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335347584:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.733085:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.733094:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.733117:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.733119:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.733122:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016133.733123:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.733162:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335347584:12345-192.168.201.116@tcp:106: Request processed in 90us (241us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.733168:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12138 00010000:00010000:3.0:1734016133.735560:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.735575:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.735587:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.735592:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.735598:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.735609:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.735804:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.735810:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.735816:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.735821:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.735826:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.735835:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.735839:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.735848:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6465 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.736539:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333847424:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.739083:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016133.739099:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335347712 00000100:00100000:3.0:1734016133.739105:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335347712 00000100:00100000:3.0:1734016133.739342:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12139 00000100:00100000:3.0:1734016133.739349:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2de450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335347712:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.739360:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.739368:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.739387:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.739389:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.739391:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016133.739393:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.739635:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2de450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335347712:12345-192.168.201.116@tcp:106: Request processed in 286us (1108us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.739641:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12139 00010000:00010000:3.0:1734016133.743078:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.743098:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.743111:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.743115:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.743120:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.743132:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.743150:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.743155:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.743160:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.743164:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.743170:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.743184:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.743189:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.743194:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e646c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.743225:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333847552:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.746802:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335347840 00000100:00100000:3.0:1734016133.746809:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335347840 00000100:00100000:3.0:1734016133.746843:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12140 00000100:00100000:3.0:1734016133.746849:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335347840:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016133.746863:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6449 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1810432->1818623) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116639 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.746926:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016133.746933:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335347840:12345-192.168.201.116@tcp:104: Request processed in 82us (260us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.746939:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12140 00000100:00100000:3.0:1734016133.759019:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335347968 00000100:00100000:3.0:1734016133.759027:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335347968 00000100:00100000:3.0:1734016133.759062:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12141 00000100:00100000:3.0:1734016133.759069:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335347968:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016133.759082:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6473 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1753088->1757183] (req 1753088->1757183) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116663 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.759098:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6473 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1753088->1757183] (req 1753088->1757183) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116663 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.759107:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6473 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1753088->1757183) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116663 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.759118:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6473 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1753088->1757183) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116663 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.759126:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6473 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1753088->1757183) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116663 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.759135:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6473 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1753088->1757183) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116663 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.759155:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366880) 00000100:00100000:3.0:1734016133.759200:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335347968:12345-192.168.201.116@tcp:105: Request processed in 131us (305us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.759207:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12141 00000100:00100000:3.0:1734016133.840738:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335348096 00000100:00100000:3.0:1734016133.840747:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335348096 00000100:00100000:3.0:1734016133.840786:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12142 00000100:00100000:3.0:1734016133.840793:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335348096:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016133.840807:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e6411 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 208896->221183) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116601 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.842768:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:3.0:1734016133.842784:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335348096:12345-192.168.201.116@tcp:104: Request processed in 1985us (2168us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.842795:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12142 00000100:00100000:3.0:1734016133.849646:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335348224 00000100:00100000:3.0:1734016133.849653:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335348224 00000100:00100000:3.0:1734016133.849700:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12143 00000100:00100000:3.0:1734016133.849707:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7698:x1818247335348224:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016133.849715:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [2953216->2961407] (req 2953216->2961407) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.849724:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [2953216->2961407] (req 2953216->2961407) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.849730:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.849737:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.849743:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.849750:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.849771:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa364900) 00000100:00100000:3.0:1734016133.849814:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7698:x1818247335348224:12345-192.168.201.116@tcp:105: Request processed in 108us (397us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.849820:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12143 00000008:00010000:3.0:1734016133.864955:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7780 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.864988:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7780 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.865004:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333848960:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016133.872086:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335348352 00000100:00100000:3.0:1734016133.872095:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335348352 00000100:00100000:3.0:1734016133.872124:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12144 00000100:00100000:3.0:1734016133.872130:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335348352:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.872142:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.872153:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.872173:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.872174:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.872177:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016133.872178:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.872214:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335348352:12345-192.168.201.116@tcp:106: Request processed in 85us (213us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.872220:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12144 00010000:00010000:3.0:1734016133.878667:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.878684:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.878698:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.878702:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.878707:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.878719:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3643390, setting rss=3643390; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.878731:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.878737:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.878743:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.878753:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.878758:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.878768:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.878772:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.878777:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6481 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.878803:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333849088:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.888171:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016133.888188:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335348480 00000100:00100000:3.0:1734016133.888198:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335348480 00000100:00100000:3.0:1734016133.888232:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12145 00000100:00100000:3.0:1734016133.888239:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2df850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335348480:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.888254:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.888272:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.888298:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.888300:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.888304:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016133.888305:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.888351:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2df850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335348480:12345-192.168.201.116@tcp:106: Request processed in 112us (302us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.888356:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12145 00010000:00010000:3.0:1734016133.900926:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.900943:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.900955:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.900961:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.900966:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.900981:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3643390, setting rss=3643390; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.900992:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.900997:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.901006:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.901012:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.901016:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.901026:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.901030:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.901035:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6488 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.902636:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333849216:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.906379:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335348608 00000100:00100000:3.0:1734016133.906388:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335348608 00000100:00100000:3.0:1734016133.906426:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12146 00000100:00100000:3.0:1734016133.906433:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335348608:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.906447:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.906456:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.906514:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.906516:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.906519:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016133.906520:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.906574:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335348608:12345-192.168.201.116@tcp:106: Request processed in 141us (428us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.906579:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12146 00010000:00010000:3.0:1734016133.919822:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.919838:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.919852:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.919859:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.919866:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.919880:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3643390, setting rss=3643390; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.920220:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.920231:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.920238:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.920245:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.920251:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.920301:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.920308:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.920313:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e648f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.921207:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333849344:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.926098:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335348736 00000100:00100000:3.0:1734016133.926107:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335348736 00000100:00100000:3.0:1734016133.926140:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12147 00000100:00100000:3.0:1734016133.926147:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335348736:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016133.926160:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.926175:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016133.926200:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016133.926202:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000020:00010000:3.0:1734016133.926205:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016133.926206:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3643390 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 64 00000100:00100000:3.0:1734016133.926249:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335348736:12345-192.168.201.116@tcp:106: Request processed in 102us (270us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016133.926255:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12147 00010000:00010000:3.0:1734016133.932802:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.932819:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.932830:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.932837:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.932842:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016133.932856:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3643390, setting rss=3643390; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016133.934228:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.934240:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.934247:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.934254:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367cc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016133.934267:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.934277:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016133.934284:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016133.934291:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6496 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016133.934343:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333849472:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016133.946301:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016133.967940:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016133.989505:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016133.999928:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335349248 00000100:00100000:3.0:1734016133.999935:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335349248 00000100:00100000:3.0:1734016133.999968:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12151 00000100:00100000:3.0:1734016133.999976:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335349248:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016133.999991:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e647a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711666a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.000042:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016134.000047:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335349248:12345-192.168.201.116@tcp:104: Request processed in 72us (760us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.000053:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12151 00000100:00100000:3.0:1734016134.017618:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016134.017632:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335349376 00000100:00100000:3.0:1734016134.017640:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335349376 00000100:00100000:3.0:1734016134.017669:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12152 00000100:00100000:3.0:1734016134.017675:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2df050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6465:x1818247335349376:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016134.017688:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.017698:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.017705:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.017715:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.017722:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.017735:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.017752:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c2f40) 00000100:00100000:3.0:1734016134.017789:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2df050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6465:x1818247335349376:12345-192.168.201.116@tcp:105: Request processed in 114us (253us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.017794:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12152 00000100:00100000:3.0:1734016134.026723:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333850496:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016134.028095:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800b6ee7a80 going to sleep for 11 seconds 00000100:00100000:3.0:1734016134.034235:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a0a80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333850880:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016134.034266:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016134.034276:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016134.034288:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016134.034295:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016134.034303:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016134.034308:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016134.034314:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1734016134.034323:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a89a0a80 x1818247333850880/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016150 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:3.0:1734016134.034331:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:3.0:1734016134.034339:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016134.034350:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:3.0:1734016134.034403:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1734016134.034407:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e64ab 00000080:00010000:3.0:1734016134.034411:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:3.0:1734016134.034413:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e64ab from it ffff8801369fbcd8 00010000:00010000:3.0:1734016134.034416:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016134.034421:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e64ab lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271166a9 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016134.034445:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016134.034454:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016134.034567:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016134.034570:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4cf0, osc lock ffff8800a8a26e40, flags 0x400001002 00000008:00010000:3.0:1734016134.034582:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016134.034584:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4d30, osc lock ffff8800a8a26d10, flags 0x400001002 00010000:00010000:3.0:1734016134.034605:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.034612:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.035178:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016134.035193:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333851008:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016134.035264:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.035271:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.035300:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333851136:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.038191:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335349504 00000100:00100000:3.0:1734016134.038198:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335349504 00000100:00100000:3.0:1734016134.038225:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335349632 00000100:00100000:3.0:1734016134.038226:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335349632 00000100:00100000:3.0:1734016134.038238:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12153 00000100:00100000:3.0:1734016134.038243:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335349504:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.038265:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e649d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116694 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.038273:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e649d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116694 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.038318:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.038320:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4022426 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 72 00000020:00010000:3.0:1734016134.038323:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.038323:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4022426 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 72 00000100:00100000:3.0:1734016134.038358:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335349504:12345-192.168.201.116@tcp:106: Request processed in 115us (365us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.038363:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12153 00000100:00100000:3.0:1734016134.038376:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12154 00000100:00100000:3.0:1734016134.038379:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335349632:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.038385:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.038391:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.038401:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.038402:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 0 00000020:00010000:3.0:1734016134.038405:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.038406:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 0 00000100:00100000:3.0:1734016134.038420:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335349632:12345-192.168.201.116@tcp:106: Request processed in 40us (345us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.038425:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12154 00010000:00010000:3.0:1734016134.041910:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.041932:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.041946:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.041951:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.041957:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.041968:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.041974:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.041980:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.041987:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.041992:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa348fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.041997:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042012:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.042018:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042024:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348fc0/0xa50fd972cb8e64b9 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.042598:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333851136:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016134.042617:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042625:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042633:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042638:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042643:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.042650:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4022426, setting rss=4022426; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.042667:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042674:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042679:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042684:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa34a1c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.042686:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042693:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.042696:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.042701:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e64b2 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.042713:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333851008:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.046850:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335349760 00000100:00100000:3.0:1734016134.046859:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335349760 00000100:00100000:3.0:1734016134.046896:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12155 00000100:00100000:3.0:1734016134.046902:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335349760:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.046913:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.046926:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.046949:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.046951:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 0 00000020:00010000:3.0:1734016134.046954:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.046955:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 0 00000100:00100000:3.0:1734016134.046997:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335349760:12345-192.168.201.116@tcp:106: Request processed in 95us (764us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.047002:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12155 00000100:00100000:3.0:1734016134.047528:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335349888 00000100:00100000:3.0:1734016134.047531:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335349888 00000100:00100000:3.0:1734016134.048071:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12156 00000100:00100000:3.0:1734016134.048076:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335349888:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.048083:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e649d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116694 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.048090:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e649d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116694 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.048104:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.048105:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4022426 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 72 00000020:00010000:3.0:1734016134.048108:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.048108:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4022426 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 72 00000100:00100000:3.0:1734016134.048227:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335349888:12345-192.168.201.116@tcp:106: Request processed in 149us (1094us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.048234:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12156 00010000:00010000:3.0:1734016134.051623:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.051638:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.051650:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.051655:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.051661:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.051674:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.051681:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.051687:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.051692:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.051697:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.051701:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.051710:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.051715:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.051721:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64c7 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.051745:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333851392:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016134.053868:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.053882:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.053892:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.053898:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.053904:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.053915:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4022426, setting rss=4022426; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.054628:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.054636:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.054643:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.054648:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.054652:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.054661:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.054665:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.054669:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e64c0 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.054694:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333851264:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.055120:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a89a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333851520:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.055173:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016134.055179:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a89a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333851648:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016134.064252:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064281:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064292:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064297:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064304:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.064317:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4022426, setting rss=4022426; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.064323:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064329:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064335:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064340:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.064345:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064359:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.064364:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064370:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e64ce lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.064407:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333851520:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016134.064790:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064798:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064804:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064809:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064814:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.064820:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.064835:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064841:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064847:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064851:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.064854:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064860:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.064863:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.064868:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e64d5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.064880:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a89a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333851648:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.065283:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333851904:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.065336:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016134.065341:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333851776:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.069686:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016134.069700:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335350272 00000100:00100000:3.0:1734016134.069707:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335350272 00000100:00100000:3.0:1734016134.069740:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12159 00000100:00100000:3.0:1734016134.069745:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dc450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335350272:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.069757:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.069766:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.069786:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.069788:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 0 00000020:00010000:3.0:1734016134.069791:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.069792:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016133 atime: 1734016133 ctime: 1734016133 blocks: 0 00000100:00100000:3.0:1734016134.069826:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dc450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335350272:12345-192.168.201.116@tcp:106: Request processed in 80us (420us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.069831:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12159 00010000:00010000:3.0:1734016134.074332:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.074347:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.074358:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.074364:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.074374:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.074386:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.074402:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.074408:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.074422:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.074429:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.074433:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.074441:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.074446:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.074451:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e64e3 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.074496:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333851904:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016134.075323:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.075332:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.075338:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.075343:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.075348:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.075356:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4022426, setting rss=4022426; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.076005:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.076014:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.076021:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.076026:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.076029:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.076039:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.076043:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.076048:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e64dc lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.076067:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333851776:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.087960:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.095358:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.102839:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335350784 00000100:00100000:3.0:1734016134.102847:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335350784 00000100:00100000:3.0:1734016134.102885:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12162 00000100:00100000:3.0:1734016134.102891:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335350784:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016134.102906:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e64a4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711669b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.102971:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016134.102976:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335350784:12345-192.168.201.116@tcp:104: Request processed in 85us (551us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.102982:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12162 00000100:00100000:3.0:1734016134.108615:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335350912 00000100:00100000:3.0:1734016134.108624:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335350912 00000100:00100000:3.0:1734016134.108660:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12163 00000100:00100000:3.0:1734016134.108665:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335350912:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016134.108679:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.108687:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.108695:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.108703:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.108709:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.108716:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.108736:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c0b40) 00000100:00100000:3.0:1734016134.108774:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335350912:12345-192.168.201.116@tcp:105: Request processed in 108us (263us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.108780:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12163 00000100:00100000:3.0:1734016134.303790:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.305481:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.306208:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 0 new 8192-byte reqbufs (1/1 left), rc = 0 00000100:00100000:3.0:1734016134.320063:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.320430:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016134.339087:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.339115:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.339131:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.339136:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.339141:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.339153:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2983201, setting rss=2983201; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.339176:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.339181:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.339187:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.339193:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.339198:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.339214:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.339219:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.339224:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e651b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.339273:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333854208:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.361846:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335351552 00000100:00100000:3.0:1734016134.361860:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335351552 00000100:00100000:3.0:1734016134.361918:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12168 00000100:00100000:3.0:1734016134.361925:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335351552:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016134.361939:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e64ea lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271166ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.362014:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016134.362019:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335351552:12345-192.168.201.116@tcp:104: Request processed in 95us (2286us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.362024:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12168 00000100:00100000:3.0:1734016134.377023:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335351680 00000100:00100000:3.0:1734016134.377031:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335351680 00000100:00100000:3.0:1734016134.377075:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12169 00000100:00100000:3.0:1734016134.377081:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335351680:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016134.377092:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6522 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5832704->5840895] (req 5832704->5840895) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116735 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.377106:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6522 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5832704->5840895] (req 5832704->5840895) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116735 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.377114:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6522 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5832704->5840895) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116735 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.377124:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6522 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5832704->5840895) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116735 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.377132:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6522 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5832704->5840895) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116735 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.377141:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6522 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5832704->5840895) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116735 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.377158:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366640) 00000100:00100000:3.0:1734016134.377202:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335351680:12345-192.168.201.116@tcp:105: Request processed in 120us (253us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.377208:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12169 00000100:00100000:3.0:1734016134.385192:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335351808 00000100:00100000:3.0:1734016134.385198:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335351808 00000100:00100000:3.0:1734016134.385238:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12170 00000100:00100000:3.0:1734016134.385244:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335351808:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.385264:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.385280:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.385304:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 730, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.385307:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.385312:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.385313:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.385356:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335351808:12345-192.168.201.116@tcp:106: Request processed in 111us (707us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.385363:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12170 00010000:00010000:3.0:1734016134.391859:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.391877:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.391888:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.391893:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.391900:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.391915:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2983201, setting rss=2983201; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.391934:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.391940:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.391946:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.391951:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.391955:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.391963:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.391967:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.391973:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6529 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.392892:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333854592:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.400920:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335351936 00000100:00100000:3.0:1734016134.400929:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335351936 00000100:00100000:3.0:1734016134.400971:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12171 00000100:00100000:3.0:1734016134.400978:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335351936:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.400992:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.401005:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.401034:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 730, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.401036:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.401040:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.401041:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.401086:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335351936:12345-192.168.201.116@tcp:106: Request processed in 107us (270us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.401093:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12171 00010000:00010000:3.0:1734016134.409159:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.409176:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.409190:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.409195:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.409201:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.409214:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2983201, setting rss=2983201; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.409234:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.409239:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.409245:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.409258:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.409262:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.409270:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.409275:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.409280:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6530 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.409316:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333854720:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.427874:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.444633:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.455270:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.459764:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333855872:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.462570:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.470914:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.482851:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.492383:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.502884:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.513981:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.546806:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.573425:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.593305:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.594018:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.618897:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016134.618911:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335353856 00000100:00100000:3.0:1734016134.618919:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335353856 00000100:00100000:3.0:1734016134.618952:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12186 00000100:00100000:3.0:1734016134.618958:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800aa984450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335353856:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.618971:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.618981:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.619005:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 733, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.619007:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.619010:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.619011:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.619050:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800aa984450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335353856:12345-192.168.201.116@tcp:106: Request processed in 92us (376us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.619057:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12186 00010000:00010000:3.0:1734016134.623688:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.623705:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.623718:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.623723:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.623730:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.623742:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5838947, setting rss=5838947; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.623750:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.623755:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.623760:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.623766:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.623770:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.623781:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.623785:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.623790:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6592 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.623820:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333858176:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016134.628051:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.628072:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.628085:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.628091:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.628098:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.628112:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2983201, setting rss=2983201; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.628122:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.628128:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.628134:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.628140:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.628145:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.628154:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.628158:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.628164:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6599 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.628193:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333858304:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016134.639818:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e657d lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3198976->3207167) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116790 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.639862:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e657d lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3198976->3207167) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116790 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.639887:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333858560:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016134.671516:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.671973:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.672000:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335354112 00000100:00100000:3.0:1734016134.672009:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335354112 00000100:00100000:3.0:1734016134.672645:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335354240 00000100:00100000:3.0:1734016134.672649:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335354240 00000100:00100000:3.0:1734016134.672663:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12187 00000100:00100000:3.0:1734016134.672672:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335354112:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.672688:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.672698:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.672730:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 736, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.672732:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.672735:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.672736:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.672782:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335354112:12345-192.168.201.116@tcp:106: Request processed in 112us (1318us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.672787:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12187 00000100:00100000:3.0:1734016134.672800:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12188 00000100:00100000:3.0:1734016134.672804:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7982d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335354240:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.672809:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e657d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3198976->3207167) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116790 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.672815:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e657d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3198976->3207167) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116790 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.672826:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 736, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.672827:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 104 00000020:00010000:3.0:1734016134.672830:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.672831:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 104 00000100:00100000:3.0:1734016134.672845:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7982d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335354240:12345-192.168.201.116@tcp:106: Request processed in 39us (872us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.672850:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12188 00000100:00100000:3.0:1734016134.689363:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016134.695170:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695188:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695200:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695206:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695212:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.695228:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2983201, setting rss=2983201; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.695236:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695251:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695257:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695263:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.695268:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695277:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.695282:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695290:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e65b5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.695331:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333859584:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016134.695350:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695356:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695362:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695369:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695374:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.695381:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5838947, setting rss=5838947; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.695388:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695394:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695399:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695405:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367cc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.695408:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695413:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.695416:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.695421:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e65ae lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.695438:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333859456:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.700879:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335354624 00000100:00100000:3.0:1734016134.700888:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335354624 00000100:00100000:3.0:1734016134.700931:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12191 00000100:00100000:3.0:1734016134.700938:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335354624:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016134.700951:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e64f1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271166f6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.701020:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016134.701025:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335354624:12345-192.168.201.116@tcp:104: Request processed in 88us (384us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.701031:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12191 00000100:00100000:3.0:1734016134.709159:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335354752 00000100:00100000:3.0:1734016134.709169:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335354752 00000100:00100000:3.0:1734016134.709210:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12192 00000100:00100000:3.0:1734016134.709216:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335354752:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016134.709227:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3780608->3788799] (req 3780608->3788799) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.709235:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3780608->3788799] (req 3780608->3788799) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.709242:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.709266:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.709273:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.709282:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.709300:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa365f80) 00000100:00100000:3.0:1734016134.709340:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335354752:12345-192.168.201.116@tcp:105: Request processed in 123us (269us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.709346:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12192 00000100:00100000:3.0:1734016134.724214:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016134.724227:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335354880 00000100:00100000:3.0:1734016134.724232:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335354880 00000100:00100000:3.0:1734016134.724275:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12193 00000100:00100000:3.0:1734016134.724281:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335354880:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.724293:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.724303:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.724326:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.724329:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.724332:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.724333:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.724373:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335354880:12345-192.168.201.116@tcp:106: Request processed in 91us (242us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.724379:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12193 00000100:00100000:3.0:1734016134.735770:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335355008 00000100:00100000:3.0:1734016134.735780:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335355008 00000100:00100000:3.0:1734016134.735835:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12194 00000100:00100000:3.0:1734016134.735841:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335355008:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.735853:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.735861:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.735886:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.735888:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.735891:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.735892:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.735943:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335355008:12345-192.168.201.116@tcp:106: Request processed in 101us (4676us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.735950:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12194 00010000:00010000:3.0:1734016134.738834:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.738852:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.738864:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.738870:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.738875:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.738888:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2983201, setting rss=2983201; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.738910:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.738915:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.738921:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.738926:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.738931:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.738942:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.738947:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.738957:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e65ca lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.738997:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333860224:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016134.744574:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e657d lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3198976->3207167) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116790 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.744610:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e657d lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3198976->3207167) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116790 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.744630:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333860480:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016134.748309:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e657d lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3198976->3207167) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116790 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.748349:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e657d lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3198976->3207167) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116790 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.748369:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333860736:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016134.751591:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335355136 00000100:00100000:3.0:1734016134.751600:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335355136 00000100:00100000:3.0:1734016134.751642:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12195 00000100:00100000:3.0:1734016134.751649:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335355136:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.751662:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.751675:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.751697:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.751699:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.751702:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.751704:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.751744:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335355136:12345-192.168.201.116@tcp:106: Request processed in 95us (240us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.751750:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12195 00010000:00010000:3.0:1734016134.753941:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.753960:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.753973:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.753980:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.753986:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.753998:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2983201, setting rss=2983201; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.754553:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.754564:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.754571:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.754578:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.754584:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.754594:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.754601:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.754608:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e65d1 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.754647:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333860864:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.755323:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333860992:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.759120:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335355264 00000100:00100000:3.0:1734016134.759129:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335355264 00000100:00100000:3.0:1734016134.759164:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12196 00000100:00100000:3.0:1734016134.759170:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335355264:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.759189:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.759200:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.759224:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.759226:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.759229:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.759230:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.759279:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335355264:12345-192.168.201.116@tcp:106: Request processed in 108us (252us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.759285:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12196 00000100:00100000:3.0:1734016134.765095:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.772165:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000020:00010000:3.0:1734016134.772639:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 7, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.772641:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 104 00000020:00010000:3.0:1734016134.772644:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.772645:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 104 00000100:00100000:3.0:1734016134.772688:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7982d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335355520:12345-192.168.201.116@tcp:106: Request processed in 295us (533us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.772694:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12198 00000100:00100000:3.0:1734016134.779554:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.785928:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.800501:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000008:00010000:3.0:1734016134.813216:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7780 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e65f4 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1888256->1896447) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116807 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.813264:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2015232 2019327) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e65f4 lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1888256->1896447) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116807 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016134.813294:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7983800 - object 0x240000400:6 offset 2015232<>2019328 00000100:00100000:3.0:1734016134.813829:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016134.813842:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333862144:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016134.816209:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:3.0:1734016134.816225:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8800b6ee76c0 00010000:00010000:3.0:1734016134.816277:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016134.816375:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7980700 - object 0x280000400:8 offset 0<>2985984 00000100:00100000:3.0:1734016134.817945:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333862400:192.168.201.116@tcp:3:fsx.0 00010000:00010000:3.0:1734016134.825293:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2019328 2023423) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e65f4 lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1888256->1896447) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116807 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016134.825324:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981f80 - object 0x240000400:6 offset 2019328<>2023424 00000100:00100000:3.0:1734016134.825344:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333862144:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016134.825365:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333862656:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016134.872443:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.872495:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016134.872518:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335356032 00000100:00100000:3.0:1734016134.872524:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335356032 00000100:00100000:3.0:1734016134.872552:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335356160 00000100:00100000:3.0:1734016134.872555:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335356160 00000100:00100000:3.0:1734016134.872565:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12202 00000100:00100000:3.0:1734016134.872571:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335356032:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.872586:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.872596:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.872618:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 732, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.872620:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.872622:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.872623:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.872664:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335356032:12345-192.168.201.116@tcp:106: Request processed in 92us (230us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.872670:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12202 00000100:00100000:3.0:1734016134.872683:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12203 00000100:00100000:3.0:1734016134.872686:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335356160:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.872691:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e65f4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1888256->1896447) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116807 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.872697:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e65f4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1888256->1896447) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116807 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.872707:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 732, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.872708:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 152 00000020:00010000:3.0:1734016134.872710:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.872712:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 152 00000100:00100000:3.0:1734016134.872723:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335356160:12345-192.168.201.116@tcp:106: Request processed in 36us (229us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.872727:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12203 00000100:00100000:3.0:1734016134.886256:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335356288 00000100:00100000:3.0:1734016134.886265:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335356288 00000100:00100000:3.0:1734016134.886300:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12204 00000100:00100000:3.0:1734016134.886306:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335356288:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016134.886322:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e65f4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1888256->1896447) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116807 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.886391:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016134.886397:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335356288:12345-192.168.201.116@tcp:104: Request processed in 90us (318us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.886402:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12204 00000100:00100000:3.0:1734016134.892837:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335356416 00000100:00100000:3.0:1734016134.892844:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335356416 00000100:00100000:3.0:1734016134.892873:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12205 00000100:00100000:3.0:1734016134.892880:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335356416:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016134.892895:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2826240->2830335] (req 2826240->2830335) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.892906:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2826240->2830335] (req 2826240->2830335) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.892913:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.892928:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.892936:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.892943:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.893082:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366d00) 00000100:00100000:3.0:1734016134.893558:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335356416:12345-192.168.201.116@tcp:105: Request processed in 677us (987us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.893565:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12205 00000100:00100000:3.0:1734016134.905585:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016134.905599:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335356544 00000100:00100000:3.0:1734016134.905606:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335356544 00000100:00100000:3.0:1734016134.905637:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12206 00000100:00100000:3.0:1734016134.905644:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff880137708850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335356544:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.905658:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.905667:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.905688:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.905690:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000020:00010000:3.0:1734016134.905693:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.905693:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000100:00100000:3.0:1734016134.905728:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff880137708850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335356544:12345-192.168.201.116@tcp:106: Request processed in 84us (268us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.905734:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12206 00010000:00010000:3.0:1734016134.915082:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.915100:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.915112:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.915117:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.915124:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.915137:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5838947, setting rss=5838947; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.915150:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.915156:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.915167:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.915173:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.915178:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.915189:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.915196:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.915202:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6610 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.919105:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333863424:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.920507:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335356672 00000100:00100000:3.0:1734016134.920514:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335356672 00000100:00100000:3.0:1734016134.920559:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12207 00000100:00100000:3.0:1734016134.920566:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335356672:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.920583:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.920594:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.920629:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.920630:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000020:00010000:3.0:1734016134.920633:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.920634:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000100:00100000:3.0:1734016134.920679:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335356672:12345-192.168.201.116@tcp:106: Request processed in 113us (289us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.920684:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12207 00010000:00010000:3.0:1734016134.927148:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.927166:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.927182:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.927187:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.927191:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.927203:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5838947, setting rss=5838947; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.927329:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.927339:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.927345:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.927351:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.927355:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.927364:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.927368:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.927373:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6617 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.929426:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333863552:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.934181:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335356800 00000100:00100000:3.0:1734016134.934192:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335356800 00000100:00100000:3.0:1734016134.934635:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12208 00000100:00100000:3.0:1734016134.934644:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335356800:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.934661:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.934677:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.934707:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.934709:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000020:00010000:3.0:1734016134.934712:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.934713:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000100:00100000:3.0:1734016134.935020:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335356800:12345-192.168.201.116@tcp:106: Request processed in 376us (1571us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.935026:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12208 00010000:00010000:3.0:1734016134.937564:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.937579:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.937589:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.937594:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.937599:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.937612:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5838947, setting rss=5838947; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.937634:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.937641:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.937646:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.937652:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.937656:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.937666:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.937670:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.937676:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e661e lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.937708:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333863680:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.938027:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333863808:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.940372:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335356928 00000100:00100000:3.0:1734016134.940380:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335356928 00000100:00100000:3.0:1734016134.940419:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12209 00000100:00100000:3.0:1734016134.940425:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335356928:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.940439:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.940447:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.940493:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.940494:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000020:00010000:3.0:1734016134.940497:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.940498:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000100:00100000:3.0:1734016134.940535:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335356928:12345-192.168.201.116@tcp:106: Request processed in 110us (260us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.940540:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12209 00010000:00010000:3.0:1734016134.943074:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.943089:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.943100:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.943106:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.943112:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.943130:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5838947, setting rss=5838947; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.943141:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.943146:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.943152:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.943158:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367cc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.943162:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.943171:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.943178:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.943183:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6625 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.943205:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333863808:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.943739:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333863936:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016134.948941:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333863936:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016134.951210:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333864064:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.955958:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335357056 00000100:00100000:3.0:1734016134.955966:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335357056 00000100:00100000:3.0:1734016134.956005:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12210 00000100:00100000:3.0:1734016134.956012:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335357056:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.956024:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.956033:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.956056:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.956058:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000020:00010000:3.0:1734016134.956061:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.956062:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000100:00100000:3.0:1734016134.956100:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335357056:12345-192.168.201.116@tcp:106: Request processed in 88us (285us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.956105:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12210 00010000:00010000:3.0:1734016134.958335:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.958350:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.958362:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.958373:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.958378:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.958392:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5838947, setting rss=5838947; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.958438:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.958443:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.958448:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.958454:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.958459:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.959171:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.959178:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.959184:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e662c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.959223:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333864064:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.966319:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335357184 00000100:00100000:3.0:1734016134.966331:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335357184 00000100:00100000:3.0:1734016134.966368:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12211 00000100:00100000:3.0:1734016134.966374:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335357184:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.966386:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.966395:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.966418:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.966419:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000020:00010000:3.0:1734016134.966422:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016134.966423:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5838947 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 160 00000100:00100000:3.0:1734016134.966492:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335357184:12345-192.168.201.116@tcp:106: Request processed in 116us (293us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.966497:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12211 00010000:00010000:3.0:1734016134.970718:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.970735:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.970745:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.970752:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.970758:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.970773:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5838947, setting rss=5838947; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.970783:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.970789:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.970795:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.970800:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.970806:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.970814:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.970818:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.970823:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6633 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.970863:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333864192:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.975959:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335357312 00000100:00100000:3.0:1734016134.975969:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335357312 00000100:00100000:3.0:1734016134.976008:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12212 00000100:00100000:3.0:1734016134.976015:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335357312:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.976029:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.976039:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.976063:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 730, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.976065:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.976068:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.976069:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.976103:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335357312:12345-192.168.201.116@tcp:106: Request processed in 89us (257us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.976108:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12212 00010000:00010000:3.0:1734016134.980110:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.980125:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.980135:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.980142:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.980148:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.980159:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2983201, setting rss=2983201; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.980180:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.980186:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.980191:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.980196:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.980201:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.980210:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.980214:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.980220:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e663a lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.981364:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333864320:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.983900:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016134.983918:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335357440 00000100:00100000:3.0:1734016134.983925:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335357440 00000100:00100000:3.0:1734016134.983968:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12213 00000100:00100000:3.0:1734016134.983975:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335357440:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016134.983989:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.983998:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016134.984024:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 730, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016134.984026:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016134.984029:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016134.984029:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016134 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016134.984071:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335357440:12345-192.168.201.116@tcp:106: Request processed in 97us (320us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.984077:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12213 00010000:00010000:3.0:1734016134.989092:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.989111:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.989129:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.989135:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.989141:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016134.989156:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2983201, setting rss=2983201; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016134.989182:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.989187:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.989194:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.989200:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016134.989205:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.989216:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016134.989221:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.989231:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6641 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016134.989269:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333864448:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016134.996346:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335357568 00000100:00100000:3.0:1734016134.996355:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335357568 00000100:00100000:3.0:1734016134.996386:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12214 00000100:00100000:3.0:1734016134.996392:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335357568:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016134.996407:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6609 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2826240->2830335) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711681c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016134.996784:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016134.996791:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335357568:12345-192.168.201.116@tcp:104: Request processed in 399us (536us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016134.996799:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12214 00000100:00100000:3.0:1734016135.009215:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335357696 00000100:00100000:3.0:1734016135.009222:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335357696 00000100:00100000:3.0:1734016135.009267:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12215 00000100:00100000:3.0:1734016135.009273:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6465:x1818247335357696:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016135.009281:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6648 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1282048->1294335] (req 1282048->1294335) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711685b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.009288:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6648 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1282048->1294335] (req 1282048->1294335) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711685b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.009301:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6648 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1282048->1294335) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711685b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.009309:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6648 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1282048->1294335) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711685b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.009316:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6648 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1282048->1294335) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711685b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.009325:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6648 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1282048->1294335) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711685b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.009339:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366880) 00000100:00100000:3.0:1734016135.009375:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6465:x1818247335357696:12345-192.168.201.116@tcp:105: Request processed in 102us (467us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.009383:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12215 00000100:00100000:3.0:1734016135.152842:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335357824 00000100:00100000:3.0:1734016135.152850:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335357824 00000100:00100000:3.0:1734016135.153112:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12216 00000100:00100000:3.0:1734016135.153120:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335357824:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.153133:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.153144:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.153172:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1754, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.153175:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016135 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016135.153178:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016135.153179:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016135 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016135.153215:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335357824:12345-192.168.201.116@tcp:106: Request processed in 95us (1148us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.153220:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12216 00000100:00100000:3.0:1734016135.159743:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335357952 00000100:00100000:3.0:1734016135.159752:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335357952 00000100:00100000:3.0:1734016135.159790:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12217 00000100:00100000:3.0:1734016135.159796:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335357952:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.159809:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.159817:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.159839:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1754, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.159841:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016135 ctime: 1734016134 blocks: 0 00000020:00010000:3.0:1734016135.159843:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016135.159844:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2983201 mtime: 1734016134 atime: 1734016135 ctime: 1734016134 blocks: 0 00000100:00100000:3.0:1734016135.159881:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335357952:12345-192.168.201.116@tcp:106: Request processed in 85us (327us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.159887:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12217 00010000:00010000:3.0:1734016135.162221:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.162268:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.162280:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.162287:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.162292:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.162304:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2983201, setting rss=2983201; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016135.162314:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.162321:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.162327:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.162336:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016135.162341:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.162355:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016135.162359:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.162365:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e665d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.162407:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333865344:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016135.169971:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.186395:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12220 00000100:00100000:3.0:1734016135.186402:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dfc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335358336:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016135.186410:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6648 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1282048->1294335) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711685b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.187040:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016135.187049:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dfc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335358336:12345-192.168.201.116@tcp:104: Request processed in 646us (1290us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.187056:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12220 00000100:00100000:3.0:1734016135.204549:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335358464 00000100:00100000:3.0:1734016135.204563:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335358464 00000100:00100000:3.0:1734016135.204605:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12221 00000100:00100000:3.0:1734016135.204612:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335358464:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016135.204629:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e6664 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711687e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.204646:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e6664 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711687e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.204656:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e6664 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711687e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.204670:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e6664 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711687e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.204681:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e6664 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711687e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.204693:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e6664 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711687e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.204728:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa364b40) 00000100:00100000:3.0:1734016135.204776:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335358464:12345-192.168.201.116@tcp:105: Request processed in 164us (2204us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.204784:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12221 00000100:00100000:3.0:1734016135.210333:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335358592 00000100:00100000:3.0:1734016135.210340:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335358592 00000100:00100000:3.0:1734016135.210378:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12222 00000100:00100000:3.0:1734016135.210384:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335358592:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016135.210396:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e65bc lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3780608->3788799) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271167cf expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.210459:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016135.210484:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335358592:12345-192.168.201.116@tcp:104: Request processed in 99us (824us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.210492:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12222 00000100:00100000:3.0:1734016135.221344:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335358720 00000100:00100000:3.0:1734016135.221351:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335358720 00000100:00100000:3.0:1734016135.221421:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12223 00000100:00100000:3.0:1734016135.221429:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335358720:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016135.221439:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e666b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116885 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.221447:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e666b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116885 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.221453:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e666b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116885 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.221461:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e666b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116885 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.221618:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e666b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116885 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.221628:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e666b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116885 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.221716:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c1680) 00000100:00100000:3.0:1734016135.221765:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335358720:12345-192.168.201.116@tcp:105: Request processed in 336us (709us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.221772:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12223 00000100:00100000:3.0:1734016135.263304:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335358848 00000100:00100000:3.0:1734016135.263313:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335358848 00000100:00100000:3.0:1734016135.263360:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12224 00000100:00100000:3.0:1734016135.263367:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335358848:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016135.263383:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e666b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116885 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.263454:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016135.263459:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335358848:12345-192.168.201.116@tcp:104: Request processed in 93us (293us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.263486:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12224 00000100:00100000:3.0:1734016135.268117:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.297289:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.304282:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.310852:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.322416:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335359488 00000100:00100000:3.0:1734016135.322425:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335359488 00000100:00100000:3.0:1734016135.322484:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12229 00000100:00100000:3.0:1734016135.322490:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335359488:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.322504:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781f600/0xa50fd972cb8e6695 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2256896->2260991) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271168bd expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.322517:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781f600/0xa50fd972cb8e6695 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2256896->2260991) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271168bd expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.322538:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.322540:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 152 00000020:00010000:3.0:1734016135.322542:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016135.322543:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 152 00000100:00100000:3.0:1734016135.322584:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335359488:12345-192.168.201.116@tcp:106: Request processed in 95us (960us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.322590:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12229 00000100:00100000:3.0:1734016135.330641:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335359616 00000100:00100000:3.0:1734016135.330650:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335359616 00000100:00100000:3.0:1734016135.330761:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12230 00000100:00100000:3.0:1734016135.330768:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335359616:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.330779:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781f600/0xa50fd972cb8e6695 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2256896->2260991) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271168bd expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.330787:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781f600/0xa50fd972cb8e6695 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2256896->2260991) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271168bd expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.330808:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.330810:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 152 00000020:00010000:3.0:1734016135.330813:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016135.330814:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 152 00000100:00100000:3.0:1734016135.330995:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335359616:12345-192.168.201.116@tcp:106: Request processed in 226us (686us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.331002:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12230 00010000:00010000:3.0:1734016135.334461:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.334547:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.334560:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.334566:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.334573:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.334587:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016135.334606:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.334614:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.334623:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.334629:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016135.334636:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.334645:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016135.334650:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.334655:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e66a3 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.334693:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333869568:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016135.338085:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781c480/0xa50fd972cb8e6680 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc271168a1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.338105:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781c480/0xa50fd972cb8e6680 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc271168a1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.338120:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333869824:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016135.341939:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7f00 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781c480/0xa50fd972cb8e6680 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc271168a1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.341967:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7f00 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781c480/0xa50fd972cb8e6680 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc271168a1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.341989:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333870080:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016135.345772:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781c480/0xa50fd972cb8e6680 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc271168a1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.345810:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781c480/0xa50fd972cb8e6680 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc271168a1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.345829:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333870336:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016135.349687:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335359744 00000100:00100000:3.0:1734016135.349697:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335359744 00000100:00100000:3.0:1734016135.349730:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12231 00000100:00100000:3.0:1734016135.349737:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335359744:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.349754:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781f600/0xa50fd972cb8e6695 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2256896->2260991) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271168bd expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.349764:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781f600/0xa50fd972cb8e6695 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2256896->2260991) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271168bd expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.349788:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.349790:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 152 00000020:00010000:3.0:1734016135.349793:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016135.349794:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 152 00000100:00100000:3.0:1734016135.349837:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335359744:12345-192.168.201.116@tcp:106: Request processed in 100us (277us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.349843:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12231 00010000:00010000:3.0:1734016135.353544:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.353561:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.353573:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.353579:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.353584:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.353596:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016135.354150:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.354156:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.354162:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.354167:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016135.354173:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.354183:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016135.354187:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.354197:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66aa lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.354240:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333870464:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016135.357033:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333870592:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016135.360703:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335359872 00000100:00100000:3.0:1734016135.360712:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335359872 00000100:00100000:3.0:1734016135.360742:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12232 00000100:00100000:3.0:1734016135.360747:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335359872:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.360760:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781f600/0xa50fd972cb8e6695 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2256896->2260991) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271168bd expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.360768:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781f600/0xa50fd972cb8e6695 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2256896->2260991) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271168bd expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.360788:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.360790:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 152 00000020:00010000:3.0:1734016135.360792:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016135.360793:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 152 00000100:00100000:3.0:1734016135.360831:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335359872:12345-192.168.201.116@tcp:106: Request processed in 82us (243us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.360835:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12232 00010000:00010000:3.0:1734016135.363048:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.363065:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.363075:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.363080:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.363085:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.363097:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016135.363357:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.363363:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.363368:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.363373:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016135.363378:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.363391:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016135.363395:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.363401:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e66b1 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.363430:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333870592:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016135.368947:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016135.368963:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335360000 00000100:00100000:3.0:1734016135.368971:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335360000 00000100:00100000:3.0:1734016135.369004:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12233 00000100:00100000:3.0:1734016135.369011:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800aa984450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335360000:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016135.369025:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781f600/0xa50fd972cb8e6695 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2256896->2260991) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271168bd expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.369085:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016135.369091:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800aa984450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335360000:12345-192.168.201.116@tcp:104: Request processed in 80us (469us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.369097:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12233 00000100:00100000:3.0:1734016135.374882:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335360128 00000100:00100000:3.0:1734016135.374891:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335360128 00000100:00100000:3.0:1734016135.374924:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12234 00000100:00100000:3.0:1734016135.374929:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335360128:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016135.374939:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66b8 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5685248->5697535] (req 5685248->5697535) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271168e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.374948:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66b8 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5685248->5697535] (req 5685248->5697535) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271168e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.374956:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66b8 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5685248->5697535) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271168e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.374963:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66b8 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5685248->5697535) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271168e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.374970:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66b8 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5685248->5697535) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271168e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.374977:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e66b8 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5685248->5697535) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271168e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.374992:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa367180) 00000100:00100000:3.0:1734016135.375029:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335360128:12345-192.168.201.116@tcp:105: Request processed in 99us (503us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.375035:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12234 00000100:00100000:3.0:1734016135.380523:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.390493:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.399691:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.407824:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.415534:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.427247:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.440919:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.444571:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335361152 00000100:00100000:3.0:1734016135.444580:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335361152 00000100:00100000:3.0:1734016135.444610:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335361280 00000100:00100000:3.0:1734016135.444613:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335361280 00000100:00100000:3.0:1734016135.444623:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12242 00000100:00100000:3.0:1734016135.444629:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335361152:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016135.444641:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781c480/0xa50fd972cb8e6680 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc271168a1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.444693:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016135.444699:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335361152:12345-192.168.201.116@tcp:104: Request processed in 71us (422us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.444705:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12242 00000100:00100000:3.0:1734016135.444716:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12243 00000100:00100000:3.0:1734016135.444719:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335361280:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016135.444725:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e6679 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2195456->2203647) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711689a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.444748:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016135.444752:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335361280:12345-192.168.201.116@tcp:104: Request processed in 32us (304us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.444756:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12243 00000100:00100000:3.0:1734016135.444834:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333872128:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016135.449488:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335361408 00000100:00100000:3.0:1734016135.449494:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335361408 00000100:00100000:3.0:1734016135.449529:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12244 00000100:00100000:3.0:1734016135.449536:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335361408:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016135.449546:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.449554:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.449560:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.449566:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.449572:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.449580:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.449598:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c2ac0) 00000100:00100000:3.0:1734016135.449636:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335361408:12345-192.168.201.116@tcp:105: Request processed in 100us (456us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.449640:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12244 00000100:00100000:3.0:1734016135.489031:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.489227:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.489272:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016135.489289:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335361536 00000100:00100000:3.0:1734016135.489295:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335361536 00000100:00100000:3.0:1734016135.489328:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335361664 00000100:00100000:3.0:1734016135.489330:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335361664 00000100:00100000:3.0:1734016135.489342:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12245 00000100:00100000:3.0:1734016135.489349:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335361536:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.489363:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e66db lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711690a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.489373:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e66db lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711690a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.489394:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.489396:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 344534 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 64 00000020:00010000:3.0:1734016135.489399:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016135.489400:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 344534 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 64 00000100:00100000:3.0:1734016135.490121:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335361536:12345-192.168.201.116@tcp:106: Request processed in 773us (1098us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.490127:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12245 00010000:00010000:3.0:1734016135.523290:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523304:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523316:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523321:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523326:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.523337:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2680125, setting rss=2680125; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016135.523342:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523346:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523351:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523355:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016135.523359:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523367:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016135.523371:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523375:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6713 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.523409:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333873664:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016135.523425:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523430:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523435:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523439:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523443:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.523447:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 344534, setting rss=344534; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016135.523458:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523462:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523489:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523493:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016135.523495:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523502:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016135.523504:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.523508:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e670c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.523517:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333873536:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016135.527644:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12252 00000100:00100000:3.0:1734016135.527651:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335362432:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.527658:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.527663:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.527693:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.527693:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2680125 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000020:00010000:3.0:1734016135.527695:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016135.527696:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2680125 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000100:00100000:3.0:1734016135.527714:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335362432:12345-192.168.201.116@tcp:106: Request processed in 63us (769us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.527719:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12252 00010000:00010000:3.0:1734016135.536287:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.536303:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.536315:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.536320:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.536325:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.536337:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2680125, setting rss=2680125; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016135.536347:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.536351:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.536356:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.536360:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016135.536365:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.536377:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016135.536380:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.536385:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6721 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.536414:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333873920:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016135.540015:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016135.540028:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335362560 00000100:00100000:3.0:1734016135.540034:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335362560 00000100:00100000:3.0:1734016135.540207:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12253 00000100:00100000:3.0:1734016135.540214:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335362560:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016135.540225:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e66db lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711690a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.540531:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016135.540537:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335362560:12345-192.168.201.116@tcp:104: Request processed in 324us (606us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.540542:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12253 00000100:00100000:3.0:1734016135.551806:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335362688 00000100:00100000:3.0:1734016135.551815:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335362688 00000100:00100000:3.0:1734016135.551849:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12254 00000100:00100000:3.0:1734016135.551855:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6465:x1818247335362688:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016135.551867:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6728 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3526656->3534847] (req 3526656->3534847) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc2711695e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.551878:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6728 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3526656->3534847] (req 3526656->3534847) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc2711695e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.551885:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6728 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3526656->3534847) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc2711695e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.551894:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6728 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3526656->3534847) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc2711695e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.551900:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6728 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3526656->3534847) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc2711695e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.551908:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6728 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3526656->3534847) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc2711695e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.551921:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366d00) 00000100:00100000:3.0:1734016135.551959:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6465:x1818247335362688:12345-192.168.201.116@tcp:105: Request processed in 105us (235us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.551964:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12254 00000100:00100000:3.0:1734016135.577300:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.588615:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333874688:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016135.591554:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.611335:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.621090:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.631145:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.641757:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.649296:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.668265:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335363712 00000100:00100000:3.0:1734016135.668272:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335363712 00000100:00100000:3.0:1734016135.672731:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12262 00000100:00100000:3.0:1734016135.672743:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335363712:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016135.672760:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6728 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3526656->3534847) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711695e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.672838:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016135.672842:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335363712:12345-192.168.201.116@tcp:104: Request processed in 103us (4702us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.672847:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12262 00000100:00100000:3.0:1734016135.684498:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335363840 00000100:00100000:3.0:1734016135.684507:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335363840 00000100:00100000:3.0:1734016135.684543:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12263 00000100:00100000:3.0:1734016135.684549:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6465:x1818247335363840:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016135.684561:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.684571:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.684578:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.684590:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.684596:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.684604:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.684629:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa367cc0) 00000100:00100000:3.0:1734016135.684667:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6465:x1818247335363840:12345-192.168.201.116@tcp:105: Request processed in 119us (641us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.684672:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12263 00000008:00010000:3.0:1734016135.897658:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### no dlm lock: extent: ffff8800b6ee7cc0 00000008:00010000:3.0:1734016135.897931:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### no dlm lock: extent: ffff8800b6ee7cc0 00000100:00100000:3.0:1734016135.897958:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333877120:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016135.905350:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335363968 00000100:00100000:3.0:1734016135.905359:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335363968 00000100:00100000:3.0:1734016135.905394:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12264 00000100:00100000:3.0:1734016135.905400:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335363968:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.905412:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.905421:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.905442:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2051, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.905444:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4337090 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 72 00000020:00010000:3.0:1734016135.905447:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016135.905449:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4337090 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 72 00000100:00100000:3.0:1734016135.905638:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335363968:12345-192.168.201.116@tcp:106: Request processed in 236us (417us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.905644:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12264 00000100:00100000:3.0:1734016135.906774:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335364096 00000100:00100000:3.0:1734016135.906778:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335364096 00000100:00100000:3.0:1734016135.906805:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12265 00000100:00100000:3.0:1734016135.906813:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335364096:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.906822:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.906829:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.906847:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2051, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.906848:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000020:00010000:3.0:1734016135.906851:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016135.906852:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000100:00100000:3.0:1734016135.906877:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335364096:12345-192.168.201.116@tcp:106: Request processed in 63us (223us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.906882:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12265 00000100:00100000:3.0:1734016135.912713:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12266 00000100:00100000:3.0:1734016135.912722:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335364224:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.912734:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.912747:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.912771:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2051, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.912772:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4337090 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 72 00000020:00010000:3.0:1734016135.912776:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016135.912777:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4337090 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 72 00000100:00100000:3.0:1734016135.912823:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335364224:12345-192.168.201.116@tcp:106: Request processed in 102us (497us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.912829:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12266 00000100:00100000:3.0:1734016135.912877:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016135.912898:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016135.912908:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335364352 00000100:00100000:3.0:1734016135.912912:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335364352 00000100:00100000:3.0:1734016135.912929:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12267 00000100:00100000:3.0:1734016135.912933:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3c450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335364352:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.912939:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.912944:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.912960:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2051, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.912961:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000020:00010000:3.0:1734016135.912963:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016135.912966:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000100:00100000:3.0:1734016135.912981:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3c450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335364352:12345-192.168.201.116@tcp:106: Request processed in 47us (106us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.912986:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12267 00000100:00100000:3.0:1734016135.921059:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335364480 00000100:00100000:3.0:1734016135.921067:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335364480 00000100:00100000:3.0:1734016135.921097:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335364608 00000100:00100000:3.0:1734016135.921099:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335364608 00000100:00100000:3.0:1734016135.921111:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12268 00000100:00100000:3.0:1734016135.921117:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335364480:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.921131:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.921140:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.921165:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2051, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.921167:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000020:00010000:3.0:1734016135.921170:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016135.921171:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000100:00100000:3.0:1734016135.921211:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335364480:12345-192.168.201.116@tcp:106: Request processed in 94us (470us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.921218:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12268 00000100:00100000:3.0:1734016135.921243:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12269 00000100:00100000:3.0:1734016135.921248:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335364608:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.921254:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.921265:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.921276:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2051, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.921277:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4337090 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 72 00000020:00010000:3.0:1734016135.921282:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016135.921283:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4337090 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 72 00000100:00100000:3.0:1734016135.921299:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335364608:12345-192.168.201.116@tcp:106: Request processed in 50us (359us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.921304:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12269 00000100:00100000:3.0:1734016135.928772:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335364736 00000100:00100000:3.0:1734016135.928780:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335364736 00000100:00100000:3.0:1734016135.929003:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12270 00000100:00100000:3.0:1734016135.929010:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335364736:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016135.929021:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6760 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271169a4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.929775:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016135.929785:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335364736:12345-192.168.201.116@tcp:104: Request processed in 774us (1769us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.929792:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12270 00000100:00100000:3.0:1734016135.948067:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335364864 00000100:00100000:3.0:1734016135.948077:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335364864 00000100:00100000:3.0:1734016135.948443:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12271 00000100:00100000:3.0:1734016135.948451:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335364864:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016135.948482:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e6798 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5558272->5570559] (req 5558272->5570559) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271169e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.948493:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e6798 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5558272->5570559] (req 5558272->5570559) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271169e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.948501:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e6798 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5558272->5570559) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271169e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.948512:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e6798 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5558272->5570559) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271169e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.948519:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e6798 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5558272->5570559) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271169e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.948528:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e6798 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5558272->5570559) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271169e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.949544:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a781f840) 00000100:00100000:3.0:1734016135.950769:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335364864:12345-192.168.201.116@tcp:105: Request processed in 2312us (3351us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.950779:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12271 00000100:00100000:3.0:1734016135.966487:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335364992 00000100:00100000:3.0:1734016135.966495:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335364992 00000100:00100000:3.0:1734016135.966530:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12272 00000100:00100000:3.0:1734016135.966536:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335364992:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.966551:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.966563:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.966589:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.966591:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000020:00010000:3.0:1734016135.966594:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016135.966595:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000100:00100000:3.0:1734016135.966633:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335364992:12345-192.168.201.116@tcp:106: Request processed in 97us (285us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.966639:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12272 00010000:00010000:3.0:1734016135.969437:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.969457:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.969498:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.969504:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.969510:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.969522:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016135.969534:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.969540:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.969546:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.969552:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367cc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016135.969556:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.969565:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016135.969570:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.969574:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e679f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.969602:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333879040:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016135.977100:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335365120 00000100:00100000:3.0:1734016135.977110:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335365120 00000100:00100000:3.0:1734016135.977146:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12273 00000100:00100000:3.0:1734016135.977152:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335365120:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.977165:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.977180:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.977207:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.977209:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000020:00010000:3.0:1734016135.977212:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016135.977214:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 1 00000100:00100000:3.0:1734016135.977289:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335365120:12345-192.168.201.116@tcp:106: Request processed in 135us (292us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.977295:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12273 00010000:00010000:3.0:1734016135.982201:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.982239:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.982254:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.982262:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.982268:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016135.982284:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016135.982296:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.982302:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.982308:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.982314:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016135.982319:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.982332:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016135.982337:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.982342:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e67a6 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016135.983512:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333879168:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016135.991412:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016135.991443:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335365248 00000100:00100000:3.0:1734016135.991453:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335365248 00000100:00100000:3.0:1734016135.991529:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12274 00000100:00100000:3.0:1734016135.991538:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dcc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335365248:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016135.991555:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e6798 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5558272->5570559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016135.991568:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e6798 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5558272->5570559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016135.991601:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016135.991603:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5569016 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 96 00000020:00010000:3.0:1734016135.991606:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016135.991606:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5569016 mtime: 1734016135 atime: 1734016135 ctime: 1734016135 blocks: 96 00000100:00100000:3.0:1734016135.991666:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dcc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335365248:12345-192.168.201.116@tcp:106: Request processed in 129us (1751us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016135.991672:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12274 00010000:00010000:3.0:1734016136.000695:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.000719:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.000734:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.000740:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.000746:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.000762:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5569016, setting rss=5569016; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016136.000795:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.000801:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.000807:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.000813:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016136.000819:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.000829:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016136.000835:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.000841:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e67ad lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.000886:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333879296:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016136.011626:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335365376 00000100:00100000:3.0:1734016136.011640:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335365376 00000100:00100000:3.0:1734016136.011704:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12275 00000100:00100000:3.0:1734016136.011712:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335365376:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016136.011730:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e6798 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5558272->5570559) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271169e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.011829:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016136.011835:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335365376:12345-192.168.201.116@tcp:104: Request processed in 124us (430us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.011842:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12275 00010000:00010000:3.0:1734016136.057724:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2093056 2097151) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016136.057795:0:10406:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7980e00 - object 0x240000400:6 offset 2093056<>2097152 00000100:00100000:3.0:1734016136.057836:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016136.057855:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333879936:192.168.201.116@tcp:4:fsx.0 00000080:00200000:3.0:1734016136.078665:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016136.078682:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016136.078699:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016136.084659:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9763320 i_blocks 104 00000080:00200000:3.0:1734016136.084674:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016136.084765:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016136.084770:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:3.0:1734016136.084786:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016136.084802:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e6767 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271169b2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016136.084813:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e6767 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271169b2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016136.084822:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:3.0:1734016136.084825:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e6767 00000080:00010000:3.0:1734016136.084831:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6767 from it ffff8801369fbcd8 00010000:00010000:3.0:1734016136.084834:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e6767 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271169b2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016136.084840:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e6767 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271169b2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016136.084858:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016136.084868:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016136.084887:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016136.084890:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4e70, osc lock ffff8800a8a271d0, flags 0x400001002 00000008:00010000:3.0:1734016136.084895:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016136.084897:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4eb0, osc lock ffff8800a8a26f70, flags 0x400001002 00010000:00010000:3.0:1734016136.084918:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b180/0xa50fd972cb8e67bb lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.084926:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b180/0xa50fd972cb8e67bb lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.084984:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349440/0xa50fd972cb8e67c2 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.084990:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349440/0xa50fd972cb8e67c2 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.089214:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.089348:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.105794:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.105857:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.118920:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.119028:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.137336:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335366400 00000100:00100000:3.0:1734016136.137344:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335366400 00000100:00100000:3.0:1734016136.137391:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12282 00000100:00100000:3.0:1734016136.137398:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335366400:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.137409:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.137418:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.137441:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1362, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.137443:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5569016 mtime: 1734016136 atime: 1734016135 ctime: 1734016136 blocks: 104 00000020:00010000:3.0:1734016136.137446:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016136.137447:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5569016 mtime: 1734016136 atime: 1734016135 ctime: 1734016136 blocks: 104 00000100:00100000:3.0:1734016136.137526:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335366400:12345-192.168.201.116@tcp:106: Request processed in 127us (309us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.137532:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12282 00000100:00100000:3.0:1734016136.139726:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335366528 00000100:00100000:3.0:1734016136.139731:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335366528 00000100:00100000:3.0:1734016136.139762:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12283 00000100:00100000:3.0:1734016136.139768:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335366528:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.139777:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.139784:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.139803:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1362, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.139804:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016135 ctime: 1734016136 blocks: 1 00000020:00010000:3.0:1734016136.139807:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016136.139808:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016135 ctime: 1734016136 blocks: 1 00000100:00100000:3.0:1734016136.139837:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335366528:12345-192.168.201.116@tcp:106: Request processed in 69us (1648us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.139841:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12283 00010000:00010000:3.0:1734016136.144382:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.144399:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.144411:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.144417:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.144423:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.144436:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016136.144446:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.144457:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.144478:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.144485:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016136.144490:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.144500:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016136.144505:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.144511:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e67ec lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.144552:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333881088:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016136.153927:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335366656 00000100:00100000:3.0:1734016136.153935:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335366656 00000100:00100000:3.0:1734016136.153964:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12284 00000100:00100000:3.0:1734016136.153971:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:8c598caf-7141-4c19-8566-0b7731516db9+5:6478:x1818247335366656:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016136.153984:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e6767 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc271169b2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016136.154053:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016136.154059:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:8c598caf-7141-4c19-8566-0b7731516db9+5:6478:x1818247335366656:12345-192.168.201.116@tcp:104: Request processed in 88us (222us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.154066:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12284 00000100:00100000:3.0:1734016136.179745:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333881728:192.168.201.116@tcp:10:fsx.0 00000080:00200000:3.0:1734016136.210412:0:9727:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016136.210418:0:9727:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016136.210454:0:9727:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1734016136.295035:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.297309:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335366912 00000100:00100000:3.0:1734016136.297316:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335366912 00000100:00100000:3.0:1734016136.297350:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12286 00000100:00100000:3.0:1734016136.297356:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335366912:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.297366:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.297374:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.297589:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1421, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.297591:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5810068 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 128 00000020:00010000:3.0:1734016136.297594:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016136.297595:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5810068 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 128 00000100:00100000:3.0:1734016136.297765:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335366912:12345-192.168.201.116@tcp:106: Request processed in 408us (991us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.297775:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12286 00000100:00100000:3.0:1734016136.313644:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016136.313673:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333883264:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016136.319024:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.319049:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335367040 00000100:00100000:3.0:1734016136.319055:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335367040 00000100:00100000:3.0:1734016136.319091:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12287 00000100:00100000:3.0:1734016136.319099:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335367040:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.319110:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.319120:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.319146:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1421, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.319148:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5810068 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 128 00000020:00010000:3.0:1734016136.319151:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016136.319152:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5810068 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 128 00000100:00100000:3.0:1734016136.319178:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335367040:12345-192.168.201.116@tcp:106: Request processed in 79us (155us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.319184:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12287 00010000:00010000:3.0:1734016136.334384:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.334400:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.334417:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.334422:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.334427:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.334440:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5810068, setting rss=5810068; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016136.334446:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.334451:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.334457:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.334488:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367cc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016136.334493:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.334502:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016136.334506:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.334517:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e680f lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.335950:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333883264:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016136.339013:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.339038:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.339053:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.339060:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.339067:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.339086:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016136.339099:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.339104:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.339117:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.339124:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016136.339130:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.339141:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016136.339148:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.339154:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6816 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.339291:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333883392:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016136.359310:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.359353:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.359373:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333883648:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016136.366038:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.366081:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.366104:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333883904:192.168.201.116@tcp:3:fsx.0 00010000:00010000:3.0:1734016136.380210:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.380235:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.380248:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.380253:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.380258:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.380272:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5810068, setting rss=5810068; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016136.380278:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.380283:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.380292:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.380297:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016136.380301:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.380311:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016136.380316:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.380320:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e681d lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.380363:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333884032:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016136.385537:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.385554:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.385564:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.385569:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.385575:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.385587:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016136.385597:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.385602:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.385608:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.385614:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016136.385618:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.385628:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016136.385633:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.385638:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6824 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.385664:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333884160:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016136.396759:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396778:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396790:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396796:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396802:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.396817:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5810068, setting rss=5810068; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016136.396824:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396830:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396836:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396842:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016136.396847:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396858:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016136.396863:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396869:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e682b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.396895:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333884288:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016136.396925:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396933:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396940:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396946:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396951:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.396958:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016136.396982:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396989:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396995:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.396999:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016136.397002:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.397008:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016136.397010:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.397016:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6832 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.397032:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333884416:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016136.403978:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335367808 00000100:00100000:3.0:1734016136.403987:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335367808 00000100:00100000:3.0:1734016136.404298:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12293 00000100:00100000:3.0:1734016136.404305:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335367808:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016136.404320:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e67b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2093056->2097151) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271169ff expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.404611:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016136.404617:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335367808:12345-192.168.201.116@tcp:104: Request processed in 313us (760us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.404623:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12293 00000100:00100000:3.0:1734016136.413709:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333884544:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016136.413740:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6839 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4583424->4587519] (req 4583424->4587519) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116a99 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.413751:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6839 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4583424->4587519] (req 4583424->4587519) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116a99 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.413758:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6839 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4583424->4587519] (req 4583424->4587519) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116a99 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.427892:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335367936 00000100:00100000:3.0:1734016136.427900:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335367936 00000100:00100000:3.0:1734016136.427935:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12294 00000100:00100000:3.0:1734016136.427941:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335367936:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016136.427953:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6839 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4583424->4587519] (req 4583424->4587519) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116a99 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.427963:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6839 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4583424->4587519] (req 4583424->4587519) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116a99 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.427971:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6839 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4583424->4587519) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116a99 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.427983:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6839 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4583424->4587519) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116a99 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.427990:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6839 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4583424->4587519) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116a99 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.427998:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6839 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4583424->4587519) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116a99 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.428018:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa365d40) 00000100:00100000:3.0:1734016136.428059:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335367936:12345-192.168.201.116@tcp:105: Request processed in 118us (249us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.428065:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12294 00000100:00100000:3.0:1734016136.517964:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.535740:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.543818:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.562708:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.571106:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.587256:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335368704 00000100:00100000:3.0:1734016136.587265:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335368704 00000100:00100000:3.0:1734016136.587346:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12300 00000100:00100000:3.0:1734016136.587355:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335368704:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016136.587370:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6839 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4583424->4587519) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116a99 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.587457:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016136.587477:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335368704:12345-192.168.201.116@tcp:104: Request processed in 111us (1424us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.587484:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12300 00000100:00100000:3.0:1734016136.612641:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335368832 00000100:00100000:3.0:1734016136.612649:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335368832 00000100:00100000:3.0:1734016136.612684:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12301 00000100:00100000:3.0:1734016136.612691:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6466:x1818247335368832:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016136.612705:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.612714:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.612722:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.612733:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.612741:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.612750:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.612774:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c2f40) 00000100:00100000:3.0:1734016136.612828:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6466:x1818247335368832:12345-192.168.201.116@tcp:105: Request processed in 137us (284us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.612834:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12301 00000100:00100000:3.0:1734016136.679912:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.679975:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.689064:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.689104:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016136.694750:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 689382 i_blocks 64 00000020:00010000:3.0:1734016136.694758:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbee6c8 (ffff88012dc12d88/0) at cl_lock_release():233 00000020:00010000:3.0:1734016136.694760:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbee6c8 (ffff88012dc12d88/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016136.694766:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7ab0 (ffff88012dc123f8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016136.694769:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7af0 (ffff8800a8a54d88/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016136.694771:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbee6c8 (ffff88012dc12d88/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016136.694773:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7ab0 (ffff88012dc12d88/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016136.694777:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7af0 (ffff88012dc12d88/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016136.694788:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016136.694797:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 689382 00000080:00200008:3.0:1734016136.696435:0:10406:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8) current size 689382, valid attrs 2068, mode 8801, uid -1227447776, gid -30720, new size 7596687, atime 1734016136.000000000, mtime 1734016136.000000000, ctime 1734016136.000000000 00010000:00010000:3.0:1734016136.696500:0:10406:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e6863 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27116ad1 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016136.696519:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781c900/0xa50fd972cb8e5f33 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115fb0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016136.696561:0:10406:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e6863 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27116ad1 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016136.696567:0:10406:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1734016136.696570:0:10406:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e6863 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27116ad1 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000020:00100000:3.0:1734016136.700968:0:10406:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff88012a5ff800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1734016136.700986:0:10406:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a7983b80 x1818247333887872/t0(0) o36->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016136.701020:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333887872:192.168.201.116@tcp:36:fsx.0 00000100:00100000:3.0:1734016136.701255:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4840 going to sleep for 11 seconds 00000100:00100000:3.0:1734016136.711239:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.760673:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335369728 00000100:00100000:3.0:1734016136.760681:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335369728 00000100:00100000:3.0:1734016136.760711:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12307 00000100:00100000:3.0:1734016136.760717:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335369728:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.760727:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.760735:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.760758:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.760760:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 0 00000020:00010000:3.0:1734016136.760763:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016136.760765:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 0 00000100:00100000:3.0:1734016136.760808:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335369728:12345-192.168.201.116@tcp:106: Request processed in 91us (1422us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.760814:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12307 00000100:00100000:3.0:1734016136.796744:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335370112 00000100:00100000:3.0:1734016136.796754:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335370112 00000100:00100000:3.0:1734016136.796791:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12310 00000100:00100000:3.0:1734016136.796799:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335370112:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.796814:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.796824:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.796849:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.796851:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 40 00000020:00010000:3.0:1734016136.796854:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016136.796855:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 40 00000100:00100000:3.0:1734016136.796896:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335370112:12345-192.168.201.116@tcp:106: Request processed in 97us (601us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.796902:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12310 00000100:00100000:3.0:1734016136.798440:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335370240 00000100:00100000:3.0:1734016136.798447:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335370240 00000100:00100000:3.0:1734016136.798778:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12311 00000100:00100000:3.0:1734016136.798786:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335370240:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.798798:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.798814:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.798836:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.798838:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000020:00010000:3.0:1734016136.798842:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016136.798843:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000100:00100000:3.0:1734016136.798881:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335370240:12345-192.168.201.116@tcp:106: Request processed in 95us (1535us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.798888:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12311 00000100:00100000:3.0:1734016136.808108:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.810685:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000020:00010000:3.0:1734016136.818404:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016136.818408:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016136.818413:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811430 (ffff8800a8a54638/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016136.818422:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.818431:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.818447:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016136.818449:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811430 (ffff8800a8a54bd8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016136.818932:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016136.818947:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016136.818964:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016136.818975:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1690, 1690] 00000008:00010000:3.0:1734016136.818994:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016136.818999:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58bab30, osc lock ffff8800a8a27300, flags 0x0 00010000:00010000:3.0:1734016136.819013:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.819024:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2727936 2732031) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.819032:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.819038:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.819044:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.819083:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8801369d4b40 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016136.819098:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134a48, count 1, dirtied 1 00000020:00010000:3.0:1734016136.819111:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016136.819113:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016136.819115:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b58bab30 (ffff8800a8a54638/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016136.819118:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.819129:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.819137:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016136.819139:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b58bab30 (ffff8800a8a54bd8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016136.819147:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016136.819192:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 1691 00000080:00200000:3.0:1734016136.819199:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016136.819204:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016136.819220:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1691, 1691] 00000008:00010000:3.0:1734016136.819225:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016136.819227:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58bab30, osc lock ffff8800a8a27300, flags 0x0 00010000:00010000:3.0:1734016136.819233:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.819240:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2732032 2736127) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.819247:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.819252:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.819258:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.823306:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016136.823310:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016136.823313:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b58bab30 (ffff8800a8a54638/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016136.823319:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.823327:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.823340:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016136.823342:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b58bab30 (ffff8800a8a54bd8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016136.823353:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016136.823364:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016136.823377:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016136.823387:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1691, 1691] 00000008:00010000:3.0:1734016136.823401:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016136.823404:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58bab30, osc lock ffff8800a8a27300, flags 0x0 00010000:00010000:3.0:1734016136.823416:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.823424:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2732032 2736127) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.823433:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.823442:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.823448:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.823622:0:10406:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8801369d4540 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.823628:0:10406:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8801369d4540 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.823633:0:10406:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8801369d4b40 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.823641:0:10406:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8801369d4b40 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016136.823654:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134a48, count 1, dirtied 1 00000020:00010000:3.0:1734016136.823665:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016136.823667:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016136.823669:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b58bab30 (ffff8800a8a54638/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016136.823671:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.823677:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.823685:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32998 (ffff8800a8a54bd8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016136.823687:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b58bab30 (ffff8800a8a54bd8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016136.823693:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016136.823717:0:10406:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8), start 6922240, end 6930431, datasync 1 00000080:00200000:3.0:1734016136.823722:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016136.823726:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1734016136.823738:0:10406:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8801369d4b40 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.823751:0:10406:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8801369d4b40 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.823906:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2727936 2732031) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e66e2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116911 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016136.823959:0:10406:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7980380 - object 0x280000400:8 offset 2727936<>2736128 00000080:00200000:3.0:1734016136.831156:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1734016136.831198:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333890688:192.168.201.116@tcp:44:fsx.0 00000100:00100000:3.0:1734016136.831253:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4b40 going to sleep for 11 seconds 00000100:00100000:3.0:1734016136.846667:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335370624 00000100:00100000:3.0:1734016136.846676:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335370624 00000100:00100000:3.0:1734016136.846722:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12314 00000100:00100000:3.0:1734016136.846729:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335370624:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.846745:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.846755:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.846779:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.846781:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000020:00010000:3.0:1734016136.846784:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016136.846786:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000100:00100000:3.0:1734016136.846828:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335370624:12345-192.168.201.116@tcp:106: Request processed in 98us (431us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.846835:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12314 00010000:00010000:3.0:1734016136.850712:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850731:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850745:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850750:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850756:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.850769:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016136.850776:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850782:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850789:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850795:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016136.850801:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850812:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016136.850818:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850824:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e68d3 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.850853:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333890944:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016136.850879:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850886:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850897:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850903:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.850908:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016136.850915:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3402383, setting rss=3402383; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016136.851667:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.851675:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.851681:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.851686:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016136.851688:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.851695:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016136.851698:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.851710:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e68da lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.851726:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333891072:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016136.856422:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016136.856437:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335370880 00000100:00100000:3.0:1734016136.856444:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335370880 00000100:00100000:3.0:1734016136.856486:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12316 00000100:00100000:3.0:1734016136.856492:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3f050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335370880:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.856508:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.856518:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.856540:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.856542:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000020:00010000:3.0:1734016136.856546:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016136.856547:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000100:00100000:3.0:1734016136.856585:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3f050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335370880:12345-192.168.201.116@tcp:106: Request processed in 93us (286us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.856596:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12316 00000100:00100000:3.0:1734016136.867012:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.876905:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016136.885740:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335371392 00000100:00100000:3.0:1734016136.885747:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335371392 00000100:00100000:3.0:1734016136.885779:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12319 00000100:00100000:3.0:1734016136.885786:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335371392:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.885798:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.885807:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.885829:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.885831:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 56 00000020:00010000:3.0:1734016136.885834:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016136.885835:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 56 00000100:00100000:3.0:1734016136.885873:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335371392:12345-192.168.201.116@tcp:106: Request processed in 87us (334us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.885880:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12319 00000100:00100000:3.0:1734016136.893304:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335371520 00000100:00100000:3.0:1734016136.893314:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335371520 00000100:00100000:3.0:1734016136.893352:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12320 00000100:00100000:3.0:1734016136.893360:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335371520:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.893376:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.893388:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.893413:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.893415:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 56 00000020:00010000:3.0:1734016136.893418:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016136.893420:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 56 00000100:00100000:3.0:1734016136.893480:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335371520:12345-192.168.201.116@tcp:106: Request processed in 103us (400us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.893488:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12320 00000100:00100000:3.0:1734016136.905706:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335371648 00000100:00100000:3.0:1734016136.905714:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335371648 00000100:00100000:3.0:1734016136.905742:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12321 00000100:00100000:3.0:1734016136.905748:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335371648:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.905761:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.905770:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.905788:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.905790:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000020:00010000:3.0:1734016136.905793:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016136.905794:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000100:00100000:3.0:1734016136.905831:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335371648:12345-192.168.201.116@tcp:106: Request processed in 83us (253us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.905836:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12321 00000100:00100000:3.0:1734016136.911896:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335371776 00000100:00100000:3.0:1734016136.911903:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335371776 00000100:00100000:3.0:1734016136.911945:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12322 00000100:00100000:3.0:1734016136.911950:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335371776:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.911965:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.911973:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.911994:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.911996:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000020:00010000:3.0:1734016136.911999:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016136.912000:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000100:00100000:3.0:1734016136.912044:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335371776:12345-192.168.201.116@tcp:106: Request processed in 92us (616us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.912051:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12322 00000100:00100000:3.0:1734016136.920299:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016136.920316:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335371904 00000100:00100000:3.0:1734016136.920323:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335371904 00000100:00100000:3.0:1734016136.920918:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12323 00000100:00100000:3.0:1734016136.920925:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2df450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335371904:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.920939:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.920949:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.920974:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.920976:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000020:00010000:3.0:1734016136.920979:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016136.920981:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000100:00100000:3.0:1734016136.921026:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2df450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335371904:12345-192.168.201.116@tcp:106: Request processed in 101us (1418us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.921033:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12323 00000100:00100000:3.0:1734016136.928657:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335372032 00000100:00100000:3.0:1734016136.928667:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335372032 00000100:00100000:3.0:1734016136.928708:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12324 00000100:00100000:3.0:1734016136.928714:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335372032:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016136.928727:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.928738:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016136.928765:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016136.928767:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000020:00010000:3.0:1734016136.928770:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016136.928771:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016136 atime: 1734016136 ctime: 1734016136 blocks: 64 00000100:00100000:3.0:1734016136.928819:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335372032:12345-192.168.201.116@tcp:106: Request processed in 103us (386us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016136.928826:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12324 00010000:00010000:3.0:1734016136.938895:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.938910:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016136.939258:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:3.0:1734016136.939278:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8801369d4000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016136.939329:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2863104 2867199) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016136.939440:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7980700 - object 0x280000400:8 offset 0<>3403776 00000100:00100000:3.0:1734016136.940778:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333892864:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016137.006110:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8801369d4000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.006387:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8801369d4000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.006413:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333892864:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016137.028147:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335372160 00000100:00100000:3.0:1734016137.028156:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335372160 00000100:00100000:3.0:1734016137.028195:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12325 00000100:00100000:3.0:1734016137.028202:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335372160:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.028229:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.028239:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.028267:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 832, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.028269:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016137 atime: 1734016136 ctime: 1734016137 blocks: 56 00000020:00010000:3.0:1734016137.028272:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016137.028273:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016137 atime: 1734016136 ctime: 1734016137 blocks: 56 00000100:00100000:3.0:1734016137.028325:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335372160:12345-192.168.201.116@tcp:106: Request processed in 123us (294us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.028331:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12325 00010000:00010000:3.0:1734016137.031900:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.031918:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.031930:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.031935:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.031940:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.031952:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3402383, setting rss=3402383; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016137.031965:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.031975:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.031981:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.031987:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016137.031991:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.032001:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016137.032007:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.032013:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6920 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.034627:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333892992:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.034980:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335372288 00000100:00100000:3.0:1734016137.034987:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335372288 00000100:00100000:3.0:1734016137.035028:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12326 00000100:00100000:3.0:1734016137.035034:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335372288:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.035047:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.035060:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.035082:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 832, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.035084:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016137 atime: 1734016136 ctime: 1734016137 blocks: 56 00000020:00010000:3.0:1734016137.035087:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016137.035088:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016137 atime: 1734016136 ctime: 1734016137 blocks: 56 00000100:00100000:3.0:1734016137.035132:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335372288:12345-192.168.201.116@tcp:106: Request processed in 97us (244us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.035139:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12326 00010000:00010000:3.0:1734016137.038430:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.038444:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.038456:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.038461:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.038485:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.038497:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3402383, setting rss=3402383; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016137.038516:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.038522:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.038528:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.038533:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016137.038538:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.038546:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016137.038550:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.038556:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6927 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.039718:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333893120:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.044949:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335372416 00000100:00100000:3.0:1734016137.044958:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335372416 00000100:00100000:3.0:1734016137.044999:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12327 00000100:00100000:3.0:1734016137.045005:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335372416:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.045020:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.045034:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.045057:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 832, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.045059:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016137 atime: 1734016136 ctime: 1734016137 blocks: 56 00000020:00010000:3.0:1734016137.045062:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016137.045062:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016137 atime: 1734016136 ctime: 1734016137 blocks: 56 00000100:00100000:3.0:1734016137.045104:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335372416:12345-192.168.201.116@tcp:106: Request processed in 99us (252us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.045111:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12327 00010000:00010000:3.0:1734016137.064121:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.064139:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.064152:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.064158:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.064164:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.064179:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3402383, setting rss=3402383; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016137.064266:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.064273:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.064279:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.064284:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016137.064290:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.064301:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016137.064305:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.064310:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e692e lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.067685:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333893248:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.068771:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.075880:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.091045:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.108829:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.127656:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.129265:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12332 00000100:00100000:3.0:1734016137.129276:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7982a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335373056:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016137.129292:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f600/0xa50fd972cb8e68ef lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2490368->2498559) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116b79 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.129704:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016137.129712:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7982a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335373056:12345-192.168.201.116@tcp:104: Request processed in 436us (2068us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.129720:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12332 00000100:00100000:3.0:1734016137.152971:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000008:00010000:3.0:1734016137.165168:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee76c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6951 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2473984->2478079) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bdb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.165214:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee76c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6951 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2473984->2478079) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bdb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.165233:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333894656:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016137.166357:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333894784:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.175944:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335373312 00000100:00100000:3.0:1734016137.175950:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335373312 00000100:00100000:3.0:1734016137.175990:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12334 00000100:00100000:3.0:1734016137.175997:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335373312:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.176006:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.176013:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e685c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ac3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.176038:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.176040:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 64 00000020:00010000:3.0:1734016137.176043:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016137.176044:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 64 00000100:00100000:3.0:1734016137.176086:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335373312:12345-192.168.201.116@tcp:106: Request processed in 89us (468us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.176092:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12334 00000100:00100000:3.0:1734016137.190288:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.199411:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333895168:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.225003:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.237980:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.248110:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000008:00010000:3.0:1734016137.404381:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.405185:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.405218:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333895808:192.168.201.116@tcp:4:fsx.0 00000080:00200008:3.0:1734016137.461325:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016137.461336:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:3.0:1734016137.461356:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016137.461373:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e688d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116b10 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016137.461390:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e688d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116b10 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016137.461400:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:3.0:1734016137.461403:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e688d 00000080:00010000:3.0:1734016137.461408:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e688d from it ffff8801369fbcd8 00010000:00010000:3.0:1734016137.461411:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e688d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116b10 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016137.461418:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e688d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116b10 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016137.461447:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016137.461460:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016137.461498:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016137.461502:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4030, osc lock ffff8800a8a27300, flags 0x400001002 00000008:00010000:3.0:1734016137.461508:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016137.461509:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4070, osc lock ffff8800a8a270a0, flags 0x400001002 00010000:00010000:3.0:1734016137.461523:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461532:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461541:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461547:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461563:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e695f lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116be9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461569:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e695f lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116be9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461576:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e695f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27116be9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461583:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e695f lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116be9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016137.461601:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7596687 i_blocks 136 00000020:00010000:3.0:1734016137.461605:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cafe8 (ffff8800a8a54bd8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016137.461608:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cafe8 (ffff8800a8a54bd8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016137.461611:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4030 (ffff8800a8a54638/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016137.461614:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4070 (ffff88012dc12d88/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016137.461617:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cafe8 (ffff8800a8a54bd8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016137.461619:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4030 (ffff8800a8a54bd8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016137.461622:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4070 (ffff8800a8a54bd8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016137.461631:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016137.461645:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016137.461649:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016137.461652:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016137.461656:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016137.461659:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4030, osc lock ffff8800a8a270a0, flags 0x400001002 00000008:00010000:3.0:1734016137.461662:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016137.461663:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4070, osc lock ffff8800a8a27300, flags 0x400001002 00010000:00010000:3.0:1734016137.461669:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461676:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461683:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461691:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461701:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e695f lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116be9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461709:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e695f lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116be9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461716:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e695f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27116be9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461722:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e695f lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116be9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016137.461732:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7596687 i_blocks 136 00000020:00010000:3.0:1734016137.461735:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cafe8 (ffff8800a8a54bd8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016137.461737:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cafe8 (ffff8800a8a54bd8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016137.461739:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4030 (ffff88012dc12d88/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016137.461741:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4070 (ffff8800a8a54638/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016137.461743:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cafe8 (ffff8800a8a54bd8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016137.461745:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4030 (ffff8800a8a54bd8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016137.461748:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4070 (ffff8800a8a54bd8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016137.461753:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016137.461758:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7596687 00000080:00200000:3.0:1734016137.461802:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016137.461805:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016137.461814:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016137.461821:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [302, 303] 00000008:00010000:3.0:1734016137.461828:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016137.461830:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58ba830, osc lock ffff8800a8a27300, flags 0x0 00010000:00010000:3.0:1734016137.461836:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461842:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1236992 1245183) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461849:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461855:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.461860:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016137.461870:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7596687 i_blocks 136 00000100:00100000:3.0:1734016137.461978:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016137.461997:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333896192:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016137.470823:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333896192:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016137.476175:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335373952 00000100:00100000:3.0:1734016137.476182:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335373952 00000100:00100000:3.0:1734016137.476220:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12339 00000100:00100000:3.0:1734016137.476226:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335373952:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.476238:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.476247:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.476268:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1023, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.476270:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 72 00000020:00010000:3.0:1734016137.476273:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016137.476274:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 72 00000100:00100000:3.0:1734016137.476306:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335373952:12345-192.168.201.116@tcp:106: Request processed in 80us (224us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.476312:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12339 00000100:00100000:3.0:1734016137.479820:0:1984:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0000-osc-ffff88012a5ff800: committing for last_committed 4294968040 gen 2 00010000:00010000:3.0:1734016137.479855:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.479873:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.479886:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.479892:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.479898:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.479911:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016137.479929:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.479936:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.479944:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.479950:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016137.479956:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.479970:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016137.479975:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.479981:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e697b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.480377:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333896320:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.485128:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335374080 00000100:00100000:3.0:1734016137.485136:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335374080 00000100:00100000:3.0:1734016137.485165:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335374208 00000100:00100000:3.0:1734016137.485167:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335374208 00000100:00100000:3.0:1734016137.485178:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12340 00000100:00100000:3.0:1734016137.485183:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335374080:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016137.485207:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e695f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27116be9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.485501:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016137.485508:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335374080:12345-192.168.201.116@tcp:104: Request processed in 324us (1049us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.485514:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12340 00000100:00100000:3.0:1734016137.491513:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333896704:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016137.499221:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333896576:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016137.502795:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016137.502811:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335374464 00000100:00100000:3.0:1734016137.502818:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335374464 00000100:00100000:3.0:1734016137.502864:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12343 00000100:00100000:3.0:1734016137.502871:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3ec50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335374464:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.502884:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.502894:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.502918:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1023, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.502920:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 72 00000020:00010000:3.0:1734016137.502923:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016137.502924:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 72 00000100:00100000:3.0:1734016137.502963:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3ec50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335374464:12345-192.168.201.116@tcp:106: Request processed in 93us (335us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.502970:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12343 00000100:00100000:3.0:1734016137.511279:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335374592 00000100:00100000:3.0:1734016137.511289:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335374592 00000100:00100000:3.0:1734016137.511390:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12344 00000100:00100000:3.0:1734016137.511396:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335374592:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.511409:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.511419:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.511445:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1023, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.511447:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 72 00000020:00010000:3.0:1734016137.511450:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016137.511451:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 72 00000100:00100000:3.0:1734016137.511665:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335374592:12345-192.168.201.116@tcp:106: Request processed in 269us (488us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.511672:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12344 00010000:00010000:3.0:1734016137.514133:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.514147:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.514157:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.514162:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.514168:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.514180:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016137.514191:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.514211:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.514218:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.514223:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016137.514227:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.514236:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016137.514241:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.514246:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6990 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.514776:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333897088:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.526056:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335374720 00000100:00100000:3.0:1734016137.526067:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335374720 00000100:00100000:3.0:1734016137.526102:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12345 00000100:00100000:3.0:1734016137.526109:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335374720:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.526122:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6982 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2564096->2576383) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c0c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.526135:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6982 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2564096->2576383) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c0c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.526158:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.526160:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 96 00000020:00010000:3.0:1734016137.526162:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016137.526163:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3402383 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 96 00000100:00100000:3.0:1734016137.526230:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335374720:12345-192.168.201.116@tcp:106: Request processed in 121us (600us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.526236:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12345 00010000:00010000:3.0:1734016137.532549:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.532565:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.532576:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.532582:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.532587:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.532600:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3402383, setting rss=3402383; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016137.533191:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.533206:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.533213:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.533218:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016137.533223:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.533233:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016137.533239:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.533244:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6997 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.533285:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333897216:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.539848:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335374848 00000100:00100000:3.0:1734016137.539856:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335374848 00000100:00100000:3.0:1734016137.539919:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12346 00000100:00100000:3.0:1734016137.539926:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335374848:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016137.539939:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6982 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2564096->2576383) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116c0c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.540124:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016137.540131:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335374848:12345-192.168.201.116@tcp:104: Request processed in 205us (516us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.540137:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12346 00000100:00100000:3.0:1734016137.548276:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335374976 00000100:00100000:3.0:1734016137.548283:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335374976 00000100:00100000:3.0:1734016137.548325:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12347 00000100:00100000:3.0:1734016137.548330:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335374976:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016137.548339:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e699e lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1961984->1966079] (req 1961984->1966079) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116c28 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.548346:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e699e lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1961984->1966079] (req 1961984->1966079) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116c28 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.548354:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e699e lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1961984->1966079) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116c28 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.548362:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e699e lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1961984->1966079) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116c28 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.548368:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e699e lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1961984->1966079) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116c28 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.548377:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e699e lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1961984->1966079) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116c28 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.549304:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366640) 00000100:00100000:3.0:1734016137.549375:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335374976:12345-192.168.201.116@tcp:105: Request processed in 1043us (1377us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.549384:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12347 00000100:00100000:3.0:1734016137.645393:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.650772:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.664986:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.671789:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.671821:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335375488 00000100:00100000:3.0:1734016137.671829:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335375488 00000100:00100000:3.0:1734016137.672639:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12351 00000100:00100000:3.0:1734016137.672651:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335375488:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.672669:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.672679:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.672706:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1854, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.672708:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 72 00000020:00010000:3.0:1734016137.672712:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016137.672713:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 72 00000100:00100000:3.0:1734016137.672760:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335375488:12345-192.168.201.116@tcp:106: Request processed in 112us (980us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.672767:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12351 00000100:00100000:3.0:1734016137.685630:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335375616 00000100:00100000:3.0:1734016137.685641:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335375616 00000100:00100000:3.0:1734016137.685745:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12352 00000100:00100000:3.0:1734016137.685753:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6479:x1818247335375616:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016137.685770:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e68b0 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27116b3a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016137.688739:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:3.0:1734016137.688751:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6479:x1818247335375616:12345-192.168.201.116@tcp:104: Request processed in 2996us (3218us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.688762:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12352 00000100:00100000:3.0:1734016137.698012:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333898880:192.168.201.116@tcp:36:fsx.0 00000080:00200000:3.0:1734016137.698053:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016137.698058:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016137.698074:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016137.698092:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1734016137.698106:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016137.698110:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d46f0, osc lock ffff8800a8a270a0, flags 0x0 00000008:00010000:3.0:1734016137.698113:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016137.698115:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4730, osc lock ffff8800a8a27690, flags 0x0 00010000:00010000:3.0:1734016137.698133:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.698145:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.698155:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.698166:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.698171:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.698179:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800ac555c00 type 11 mode 2 res 10737419264/8 (0 18446744073709551615) 00010000:00010000:3.0:1734016137.698188:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349440/0xa50fd972cb8e69c8 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.698204:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349440/0xa50fd972cb8e69c8 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.698228:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349440/0xa50fd972cb8e69c8 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.698244:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333899136:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.698279:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4300 going to sleep for 11 seconds 00000100:00100000:3.0:1734016137.710304:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016137.710319:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333899264:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016137.716404:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333899264:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016137.729710:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016137.737956:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333899392:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016137.764003:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016137.764019:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335376256 00000100:00100000:3.0:1734016137.764026:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335376256 00000100:00100000:3.0:1734016137.764057:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12357 00000100:00100000:3.0:1734016137.764065:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3f850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335376256:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.764079:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349440/0xa50fd972cb8e69c8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c59 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.764089:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349440/0xa50fd972cb8e69c8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c59 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.764113:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1023, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.764115:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 96 00000020:00010000:3.0:1734016137.764118:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016137.764119:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 96 00000100:00100000:3.0:1734016137.764161:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3f850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335376256:12345-192.168.201.116@tcp:106: Request processed in 97us (281us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.764168:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12357 00000100:00100000:3.0:1734016137.773364:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335376384 00000100:00100000:3.0:1734016137.773374:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335376384 00000100:00100000:3.0:1734016137.773423:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12358 00000100:00100000:3.0:1734016137.773432:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335376384:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016137.773449:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349440/0xa50fd972cb8e69c8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116c59 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.773805:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016137.773813:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335376384:12345-192.168.201.116@tcp:104: Request processed in 382us (567us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.773822:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12358 00000100:00100000:3.0:1734016137.783421:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335376512 00000100:00100000:3.0:1734016137.783430:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335376512 00000100:00100000:3.0:1734016137.783694:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12359 00000100:00100000:3.0:1734016137.783703:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335376512:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016137.783717:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [757760->761855] (req 757760->761855) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.783726:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [757760->761855] (req 757760->761855) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.783733:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.783742:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.783831:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.783840:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.783851:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a781d680) 00000100:00100000:3.0:1734016137.783887:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335376512:12345-192.168.201.116@tcp:105: Request processed in 186us (548us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.783893:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12359 00000008:00010000:3.0:1734016137.828813:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a7b190c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.828843:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a7b190c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.828860:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333900928:192.168.201.116@tcp:4:fsx.0 00000100:00100000:3.0:1734016137.845246:0:1983:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff88012a5ff800: committing for last_committed 4294969516 gen 1 00000100:00100000:3.0:1734016137.845303:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333901056:192.168.201.116@tcp:16:fsx.0 00000100:00100000:3.0:1734016137.855943:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335377408 00000100:00100000:3.0:1734016137.855956:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335377408 00000100:00100000:3.0:1734016137.855998:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12360 00000100:00100000:3.0:1734016137.856005:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335377408:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.856023:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.856034:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.856061:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.856063:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 112 00000020:00010000:3.0:1734016137.856066:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016137.856067:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 112 00000100:00100000:3.0:1734016137.856117:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335377408:12345-192.168.201.116@tcp:106: Request processed in 111us (519us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.856123:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12360 00000100:00100000:3.0:1734016137.859632:0:1984:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff88012a571000: committing for last_committed 4294969515 gen 2 00010000:00010000:3.0:1734016137.859678:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.859690:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.859701:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.859707:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.859712:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.859727:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016137.859739:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.859744:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.859750:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.859755:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016137.859760:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.859771:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016137.859776:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.859781:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e69eb lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.860329:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333901184:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.868009:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335377536 00000100:00100000:3.0:1734016137.868018:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335377536 00000100:00100000:3.0:1734016137.868179:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12361 00000100:00100000:3.0:1734016137.868188:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335377536:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.868210:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.868220:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.868244:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.868246:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 112 00000020:00010000:3.0:1734016137.868250:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016137.868251:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 112 00000100:00100000:3.0:1734016137.868432:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335377536:12345-192.168.201.116@tcp:106: Request processed in 244us (547us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.868442:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12361 00010000:00010000:3.0:1734016137.874987:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.875009:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.875027:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.875033:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.875039:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.875054:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016137.875067:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.875073:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.875080:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.875086:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016137.875093:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.875105:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016137.875112:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.875117:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e69f2 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.875165:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333901312:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016137.890427:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.890545:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.890569:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333901568:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016137.899263:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.899291:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.899311:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333901824:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016137.900024:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333901952:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.902131:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335377792 00000100:00100000:3.0:1734016137.902139:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335377792 00000100:00100000:3.0:1734016137.902172:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12362 00000100:00100000:3.0:1734016137.902179:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335377792:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.902200:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.902209:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.902232:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.902234:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 112 00000020:00010000:3.0:1734016137.902237:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016137.902238:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 112 00000100:00100000:3.0:1734016137.902271:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335377792:12345-192.168.201.116@tcp:106: Request processed in 92us (223us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.902277:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12362 00010000:00010000:3.0:1734016137.906131:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.906148:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.906160:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.906167:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.906173:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.906188:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016137.906209:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.906216:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.906222:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.906227:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016137.906233:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.906241:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016137.906247:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.906252:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e69f9 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.910851:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333901952:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.914345:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335377920 00000100:00100000:3.0:1734016137.914354:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335377920 00000100:00100000:3.0:1734016137.914394:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12363 00000100:00100000:3.0:1734016137.914401:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335377920:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016137.914415:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.914423:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016137.914446:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016137.914448:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 112 00000020:00010000:3.0:1734016137.914452:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016137.914453:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016137 atime: 1734016137 ctime: 1734016137 blocks: 112 00000100:00100000:3.0:1734016137.914524:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335377920:12345-192.168.201.116@tcp:106: Request processed in 122us (281us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.914530:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12363 00010000:00010000:3.0:1734016137.920203:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.920218:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.920230:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.920236:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.920242:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016137.920255:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016137.920266:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.920272:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.920278:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.920284:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016137.920288:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.920298:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016137.920302:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.920307:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6a00 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.922400:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333902080:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016137.940218:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016137.940238:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335378048 00000100:00100000:3.0:1734016137.940247:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335378048 00000100:00100000:3.0:1734016137.940287:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12364 00000100:00100000:3.0:1734016137.940295:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dcc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335378048:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016137.940311:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6974 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3698688->3702783) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116bfe expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.940365:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016137.940369:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dcc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335378048:12345-192.168.201.116@tcp:104: Request processed in 76us (1436us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.940375:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12364 00000100:00100000:3.0:1734016137.954843:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333902208:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016137.954880:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1163264->1175551] (req 1163264->1175551) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.954893:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1163264->1175551] (req 1163264->1175551) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.954900:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1163264->1175551] (req 1163264->1175551) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016137.957964:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335378176 00000100:00100000:3.0:1734016137.957973:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335378176 00000100:00100000:3.0:1734016137.958026:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12365 00000100:00100000:3.0:1734016137.958033:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335378176:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016137.958047:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1163264->1175551] (req 1163264->1175551) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.958059:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1163264->1175551] (req 1163264->1175551) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.958066:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.958077:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.958085:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.958094:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016137.958126:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa367cc0) 00000100:00100000:3.0:1734016137.958175:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335378176:12345-192.168.201.116@tcp:105: Request processed in 142us (1038us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016137.958181:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12365 00000008:00010000:3.0:1734016138.079135:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8800b6ee7900 00010000:00010000:3.0:1734016138.079261:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016138.079439:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7980000 - object 0x280000400:8 offset 0<>4194304 00000100:00100000:3.0:1734016138.079987:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016138.079999:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333902848:192.168.201.116@tcp:3:fsx.0 00010000:00010000:3.0:1734016138.085918:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.085940:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4194304 4198399) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.088424:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.088437:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.088617:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8801369d49c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.088657:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4194304 4198399) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016138.088714:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7983100 - object 0x240000400:6 offset 4194304<>6037504 00000100:00100000:3.0:1734016138.088731:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016138.088742:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333903104:192.168.201.116@tcp:3:fsx.0 00000020:00010000:3.0:1734016138.104103:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32038 (ffff8800a8a54008/0) at cl_lock_release():233 00000020:00010000:3.0:1734016138.104113:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32038 (ffff8800a8a54008/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.104117:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811f30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016138.104126:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.104138:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.104154:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32038 (ffff8800a8a54008/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.104157:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811f30 (ffff8800a8a54008/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016138.104163:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 5528 bytes 00000080:00200000:3.0:1734016138.104177:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016138.104180:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 5528, restart: 0 00000080:00200000:3.0:1734016138.104197:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 5528 00000080:00200008:3.0:1734016138.104220:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016138.104225:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016138.104243:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016138.104256:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:3.0:1734016138.104294:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:3.0:1734016138.104301:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:3.0:1734016138.104309:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:3.0:1734016138.104330:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333903232:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.104400:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4000 going to sleep for 11 seconds 00000100:00100000:3.0:1734016138.121529:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333903232:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.121552:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016138.121566:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016138.121577:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016138.121587:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016138.121598:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016138.121604:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1734016138.121610:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1734016138.121620:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a7982a00 x1818247333903232/t0(0) o101->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016154 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:3.0:1734016138.121629:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:3.0:1734016138.121641:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.121652:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:3.0:1734016138.121703:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1734016138.121708:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e6a0e 00000080:00010000:3.0:1734016138.121712:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:3.0:1734016138.121715:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6a0e from it ffff8801369fbcd8 00010000:00010000:3.0:1734016138.121718:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.121724:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016138.121748:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016138.121758:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016138.121776:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.121779:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4030, osc lock ffff8800a8a27560, flags 0x400001002 00000008:00010000:3.0:1734016138.121784:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.121785:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4070, osc lock ffff8800a8a27430, flags 0x400001002 00010000:00010000:3.0:1734016138.121797:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121806:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121814:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121820:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 5/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121832:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121843:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121849:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121854:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016138.121868:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10227755 i_blocks 184 00000020:00010000:3.0:1734016138.121872:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c9240 (ffff8800a8a54008/0) at cl_lock_release():233 00000020:00010000:3.0:1734016138.121874:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c9240 (ffff8800a8a54008/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.121876:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4030 (ffff8800a8a546c8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.121879:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4070 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.121881:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c9240 (ffff8800a8a54008/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.121882:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4030 (ffff8800a8a54008/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.121885:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4070 (ffff8800a8a54008/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016138.121891:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016138.121906:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016138.121910:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016138.121912:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016138.121916:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.121917:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4030, osc lock ffff8800a8a27430, flags 0x400001002 00000008:00010000:3.0:1734016138.121919:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.121920:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4070, osc lock ffff8800a8a27560, flags 0x400001002 00010000:00010000:3.0:1734016138.121924:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121931:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121937:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121942:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 5/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121949:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121955:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121961:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.121965:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016138.121973:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10227755 i_blocks 184 00000020:00010000:3.0:1734016138.121975:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c9240 (ffff8800a8a54008/0) at cl_lock_release():233 00000020:00010000:3.0:1734016138.121977:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c9240 (ffff8800a8a54008/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.121979:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4030 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.121981:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4070 (ffff8800a8a546c8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.121983:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c9240 (ffff8800a8a54008/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.121984:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4030 (ffff8800a8a54008/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.121986:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4070 (ffff8800a8a54008/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016138.121990:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016138.121994:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10227755 00000080:00200008:3.0:1734016138.122051:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 4740020, type: SEEK_SET 00000080:00200008:3.0:1734016138.122054:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 4740020, type: SEEK_SET, rc = 4740020 00000080:00200008:3.0:1734016138.122061:0:10406:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 4740020, count: 7208 00000080:00200000:3.0:1734016138.122067:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 4740020, bytes: 7208 00000080:00200000:3.0:1734016138.122071:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1734016138.122076:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5ce10[0x4, 2499, [0x200000401:0xc:0x0] hash] 00000020:00200000:3.0:1734016138.122078:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [4740020, 4747228) 0 0 00000080:00200000:3.0:1734016138.122080:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1734016138.122087:0:10406:0:(file.c:1893:ll_file_io_generic()) Range lock [1157, 1158] 00020000:00200000:3.0:1734016138.122093:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [4740020, 4747228) 4747228, 7208 00020000:00200000:3.0:1734016138.122096:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016138.122106:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1157, 1158] 00000008:00010000:3.0:1734016138.122112:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016138.122114:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58bab30, osc lock ffff8800a8a27560, flags 0x0 00010000:00010000:3.0:1734016138.122120:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.122126:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (544768 552959) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.122132:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.122137:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.122143:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016138.122151:0:10406:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [4740020, 4747228) 00000080:00200000:3.0:1734016138.122176:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 1157 of 948 to 3148 bytes 00000080:00200000:3.0:1734016138.193421:0:10406:0:(rw26.c:1006:ll_write_end()) pos 4740020, len 3148, copied 3148 00000080:00200000:3.0:1734016138.193428:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff88012d4e15e8[3 ffff8800b3a5ce90 2 1 ffff8800a79c92a0] 00000080:00200000:3.0:1734016138.193431:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea00049931c0 2fffff00000825 2:0 ffff88012d4e15e8 1157 lru 00000080:00200000:3.0:1734016138.193437:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff88012d4e1648 133: 1< 1 - - > 2< 544768 0 4096 0x0 0x9 | ffff8800a9d60708 ffff8800aa37b1e0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1734016138.193438:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff88012d4e15e8 00000080:00200000:3.0:1734016138.193439:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:3.0:1734016138.193443:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 1158 of 0 to 4060 bytes 00000080:00200000:3.0:1734016138.193449:0:10406:0:(rw26.c:1006:ll_write_end()) pos 4743168, len 4060, copied 4060 00000080:00200000:3.0:1734016138.193451:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff88012d4e16c0[3 ffff8800b3a5ce90 2 1 ffff8800a79c92a0] 00000080:00200000:3.0:1734016138.193453:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea00049cf100 2fffff00000825 2:0 ffff88012d4e16c0 1158 lru 00000080:00200000:3.0:1734016138.193457:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff88012d4e1720 134: 1< 1 - - > 2< 548864 0 4096 0x0 0x9 | ffff8800a9d60708 ffff8800aa37b1e0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1734016138.193458:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff88012d4e16c0 00000080:00200000:3.0:1734016138.193459:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:3.0:1734016138.193485:0:10406:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 2, from 948, to 4060 00000008:00010000:3.0:1734016138.193512:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8801369d4000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.193532:0:10406:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8801369d4000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016138.193542:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134a48, count 2, dirtied 2 00000080:00200000:3.0:1734016138.193546:0:10406:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 2 pages 7208 bytes, tot: 7208 00000080:00200000:3.0:1734016138.193548:0:10406:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 7208 00000020:00010000:3.0:1734016138.193563:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32038 (ffff8800a8a54008/0) at cl_lock_release():233 00000020:00010000:3.0:1734016138.193566:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32038 (ffff8800a8a54008/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.193569:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b58bab30 (ffff8800a8a546c8/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016138.193576:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.193584:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.193600:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32038 (ffff8800a8a54008/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.193602:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b58bab30 (ffff8800a8a54008/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016138.193607:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 7208 bytes 00000080:00200000:3.0:1734016138.193613:0:10406:0:(file.c:1930:ll_file_io_generic()) Range unlock [1157, 1158] 00000080:00200000:3.0:1734016138.193621:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016138.193623:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 7208, restart: 0 00000080:00200000:3.0:1734016138.193632:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10227755 i_blocks 184 00000080:00200000:3.0:1734016138.193636:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 7208 00000080:00200008:3.0:1734016138.193654:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016138.193659:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:3.0:1734016138.193676:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016138.193691:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781cb40/0xa50fd972cb8e69cf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116c67 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.193700:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781cb40/0xa50fd972cb8e69cf lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116c67 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016138.193708:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:3.0:1734016138.193710:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e69cf 00000080:00010000:3.0:1734016138.193715:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e69cf from it ffff8801369fbcd8 00010000:00010000:3.0:1734016138.193717:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781cb40/0xa50fd972cb8e69cf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116c67 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.193722:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781cb40/0xa50fd972cb8e69cf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116c67 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016138.193735:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016138.193746:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016138.193764:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.193767:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4930, osc lock ffff8800a8a27560, flags 0x400001002 00000008:00010000:3.0:1734016138.193771:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.193772:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4970, osc lock ffff8800a8a27430, flags 0x400001002 00010000:00010000:3.0:1734016138.193790:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.193797:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.193848:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016138.193861:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333903360:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.193919:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.193926:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.193951:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333903488:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016138.202864:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8801369d49c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.203012:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8801369d49c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6a07 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1163264->1175551) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.203039:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333903104:192.168.201.116@tcp:3:fsx.0 00010000:00010000:3.0:1734016138.208861:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.208879:0:1984:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.208896:0:1984:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.208907:0:1984:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.208922:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 6033451, setting rss=6033451; kms=6033451, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.208933:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.208940:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.208950:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.208957:0:1984:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.208973:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333903360:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.213874:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.213890:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.213902:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.213908:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.213914:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.213929:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.213973:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.213979:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.213986:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.213992:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa34a880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.213996:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.214005:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.214009:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.214014:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e6a1c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.215844:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333903488:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.219834:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335378560 00000100:00100000:3.0:1734016138.219842:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335378560 00000100:00100000:3.0:1734016138.219892:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12367 00000100:00100000:3.0:1734016138.219898:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335378560:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.219910:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.219923:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.219946:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2499, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.219948:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 112 00000020:00010000:3.0:1734016138.219951:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.219952:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 112 00000100:00100000:3.0:1734016138.220127:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335378560:12345-192.168.201.116@tcp:106: Request processed in 228us (588us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.220133:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12367 00000100:00100000:3.0:1734016138.227652:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335378688 00000100:00100000:3.0:1734016138.227661:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335378688 00000100:00100000:3.0:1734016138.227693:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12368 00000100:00100000:3.0:1734016138.227701:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335378688:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.227713:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.227722:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.227743:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2499, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.227745:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 112 00000020:00010000:3.0:1734016138.227748:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.227749:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 112 00000100:00100000:3.0:1734016138.227789:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335378688:12345-192.168.201.116@tcp:106: Request processed in 89us (520us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.227795:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12368 00000100:00100000:3.0:1734016138.234600:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12369 00000100:00100000:3.0:1734016138.234608:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335378816:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016138.234624:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e69e4 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 757760->761855) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116c7c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.234938:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016138.234944:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335378816:12345-192.168.201.116@tcp:104: Request processed in 337us (880us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.234951:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12369 00000100:00100000:3.0:1734016138.252500:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016138.252515:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335378944 00000100:00100000:3.0:1734016138.252522:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335378944 00000100:00100000:3.0:1734016138.252565:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12370 00000100:00100000:3.0:1734016138.252572:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2df050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335378944:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016138.252583:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3346432->3350527] (req 3346432->3350527) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.252593:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3346432->3350527] (req 3346432->3350527) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.252600:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.252609:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.252621:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.252629:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.252654:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c21c0) 00000100:00100000:3.0:1734016138.252693:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2df050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335378944:12345-192.168.201.116@tcp:105: Request processed in 121us (858us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.252700:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12370 00000100:00100000:3.0:1734016138.264018:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333904128:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016138.357603:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335379072 00000100:00100000:3.0:1734016138.357610:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335379072 00000100:00100000:3.0:1734016138.357641:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12371 00000100:00100000:3.0:1734016138.357647:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335379072:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.357659:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.357668:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.357688:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.357690:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 152 00000020:00010000:3.0:1734016138.357692:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.357693:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 152 00000100:00100000:3.0:1734016138.357728:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335379072:12345-192.168.201.116@tcp:106: Request processed in 81us (311us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.357734:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12371 00000100:00100000:3.0:1734016138.361143:0:1984:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff88012a5ff800: committing for last_committed 4294969517 gen 1 00010000:00010000:3.0:1734016138.361178:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.361213:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.361230:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.361236:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.361241:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.361254:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.361263:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.361268:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.361274:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.361281:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.361289:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.361299:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.361304:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.361309:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6a38 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.361343:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333904896:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.362279:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016138.362290:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333905024:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.369368:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335379200 00000100:00100000:3.0:1734016138.369376:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335379200 00000100:00100000:3.0:1734016138.369405:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12372 00000100:00100000:3.0:1734016138.369411:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335379200:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.369422:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.369431:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.370056:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.370059:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 152 00000020:00010000:3.0:1734016138.370062:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.370063:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 152 00000100:00100000:3.0:1734016138.370103:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335379200:12345-192.168.201.116@tcp:106: Request processed in 691us (1001us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.370110:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12372 00010000:00010000:3.0:1734016138.382272:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.382290:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.382302:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.382311:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.382318:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.382330:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.382340:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.382345:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.382360:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.382366:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.382372:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.382383:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.382387:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.382393:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6a3f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.382431:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333905024:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016138.395629:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.395666:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.395685:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333905152:192.168.201.116@tcp:4:fsx.0 00000100:00100000:3.0:1734016138.401450:0:1983:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff88012a571000: committing for last_committed 4294969519 gen 2 00000100:00100000:3.0:1734016138.401527:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333905280:192.168.201.116@tcp:16:fsx.0 00000100:00100000:3.0:1734016138.412002:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335379328 00000100:00100000:3.0:1734016138.412011:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335379328 00000100:00100000:3.0:1734016138.412046:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12373 00000100:00100000:3.0:1734016138.412052:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335379328:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.412065:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.412075:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.412100:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.412102:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 168 00000020:00010000:3.0:1734016138.412105:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.412106:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 168 00000100:00100000:3.0:1734016138.412149:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335379328:12345-192.168.201.116@tcp:106: Request processed in 96us (252us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.412156:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12373 00010000:00010000:3.0:1734016138.416158:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.416175:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.416197:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.416202:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.416209:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.416223:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.416287:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.416294:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.416301:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.416307:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.416311:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.416321:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.416326:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.416331:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6a46 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.419550:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333905408:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.426670:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335379456 00000100:00100000:3.0:1734016138.426677:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335379456 00000100:00100000:3.0:1734016138.426707:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12374 00000100:00100000:3.0:1734016138.426714:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335379456:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.426727:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.426736:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.426758:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.426760:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 168 00000020:00010000:3.0:1734016138.426762:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.426763:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 168 00000100:00100000:3.0:1734016138.426800:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335379456:12345-192.168.201.116@tcp:106: Request processed in 87us (274us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.426806:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12374 00010000:00010000:3.0:1734016138.430991:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.431007:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.431020:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.431025:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.431031:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.431043:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.432892:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.432904:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.432913:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.432919:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.432925:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.432935:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.432941:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.432947:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6a4d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.434879:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333905536:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.473609:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.473625:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.474045:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:3.0:1734016138.474062:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8801369d4a80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.474113:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4751360 4755455) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016138.474246:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981c00 - object 0x240000400:6 offset 0<>4755456 00000100:00100000:3.0:1734016138.474270:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333905792:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016138.474370:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:3.0:1734016138.474375:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8801369d4840 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.474398:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4190208 4194303) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016138.474440:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982a00 - object 0x240000400:6 offset 4190208<>6037504 00000100:00100000:3.0:1734016138.474452:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333906048:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016138.557805:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8801369d4a80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.563766:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8801369d4a80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6a15 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116cb4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.563803:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333905792:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016138.571221:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335379584 00000100:00100000:3.0:1734016138.571229:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335379584 00000100:00100000:3.0:1734016138.571812:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12375 00000100:00100000:3.0:1734016138.571819:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6478:x1818247335379584:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016138.571835:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6a0e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27116cad expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.575199:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016138.575216:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6478:x1818247335379584:12345-192.168.201.116@tcp:104: Request processed in 3390us (4943us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.575228:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12375 00000100:00100000:3.0:1734016138.588366:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.588409:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.619252:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.619274:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335379968 00000100:00100000:3.0:1734016138.619279:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335379968 00000100:00100000:3.0:1734016138.619308:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12378 00000100:00100000:3.0:1734016138.619314:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335379968:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016138.619322:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.619334:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.619341:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.619349:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.619356:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.619364:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.619387:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c2f40) 00000100:00100000:3.0:1734016138.619422:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335379968:12345-192.168.201.116@tcp:105: Request processed in 108us (172us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.619429:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12378 00000080:00200000:3.0:1734016138.652642:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016138.652654:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016138.652671:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.652674:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d49f0, osc lock ffff8800a8a27a20, flags 0x400001002 00000008:00010000:3.0:1734016138.652697:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.652698:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4a30, osc lock ffff8800a8a27b50, flags 0x400001002 00010000:00010000:3.0:1734016138.652718:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34bcc0/0xa50fd972cb8e6a62 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.652725:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34bcc0/0xa50fd972cb8e6a62 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.652764:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016138.652773:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333907328:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.653259:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.653266:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.653292:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333907456:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.660559:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.667687:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335380224 00000100:00100000:3.0:1734016138.667696:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335380224 00000100:00100000:3.0:1734016138.667731:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12380 00000100:00100000:3.0:1734016138.667738:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335380224:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.667750:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.667758:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.667809:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 626, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.667811:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2557966 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 96 00000020:00010000:3.0:1734016138.667816:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.667817:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2557966 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 96 00000100:00100000:3.0:1734016138.667863:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335380224:12345-192.168.201.116@tcp:106: Request processed in 125us (274us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.667869:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12380 00010000:00010000:3.0:1734016138.676061:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.676083:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.676098:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.676105:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.676112:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.676127:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2557966, setting rss=2557966; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.676140:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.676147:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.676158:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.676164:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa348900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.676170:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.676191:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.676200:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.676206:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e6a69 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.676235:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333907456:192.168.201.116@tcp:101:fsx.0 00000080:00200000:3.0:1734016138.688006:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6752270 i_blocks 168 00000020:00010000:3.0:1734016138.688013:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b5a11918 (ffff88012bf7f1b8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016138.688016:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b5a11918 (ffff88012bf7f1b8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.688022:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a550f0 (ffff88012bf7f098/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.688025:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55130 (ffff88012bf7f008/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.688028:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b5a11918 (ffff88012bf7f1b8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.688030:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a550f0 (ffff88012bf7f1b8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.688034:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55130 (ffff88012bf7f1b8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016138.688049:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016138.688062:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 6752270 00000080:00200008:3.0:1734016138.688175:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 6834965, type: SEEK_SET 00000080:00200008:3.0:1734016138.688192:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 6834965, type: SEEK_SET, rc = 6834965 00000080:00200008:3.0:1734016138.688206:0:10406:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 6834965, count: 7591 00000080:00200000:3.0:1734016138.688215:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 6834965, bytes: 7591 00000080:00200000:3.0:1734016138.688223:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1734016138.688231:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5cd48[0x4, 626, [0x200000401:0xc:0x0] hash] 00000020:00200000:3.0:1734016138.688233:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [6834965, 6842556) 0 0 00000080:00200000:3.0:1734016138.688237:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1734016138.688253:0:10406:0:(file.c:1893:ll_file_io_generic()) Range lock [1668, 1670] 00020000:00200000:3.0:1734016138.688264:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [6834965, 6842556) 6842556, 7591 00020000:00200000:3.0:1734016138.688268:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016138.688290:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1668, 1670] 00000008:00010000:3.0:1734016138.688309:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016138.688314:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58bab30, osc lock ffff8800a8a277c0, flags 0x0 00010000:00010000:3.0:1734016138.688337:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.688350:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2637824 2650111) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.688359:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.688365:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.688371:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016138.688379:0:10406:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [6834965, 6842556) 00000080:00200000:3.0:1734016138.688396:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 1668 of 2837 to 1259 bytes 00000080:00200000:3.0:1734016138.688483:0:10406:0:(rw26.c:1006:ll_write_end()) pos 6834965, len 1259, copied 1259 00000080:00200000:3.0:1734016138.688490:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a9d3fca8[3 ffff8800b3a5cdc8 2 1 ffff8800b5a11978] 00000080:00200000:3.0:1734016138.688492:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea00049ab800 2fffff00000805 3:0 ffff8800a9d3fca8 1668 lru 00000080:00200000:3.0:1734016138.688499:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a9d3fd08 644: 1< 0 - - > 2< 2637824 0 0 0x0 0x0 | ffff8800aa960708 ffff8800a90061a0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1734016138.688501:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a9d3fca8 00000080:00200000:3.0:1734016138.688502:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:3.0:1734016138.688505:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 1669 of 0 to 4096 bytes 00000080:00200000:3.0:1734016138.688519:0:10406:0:(rw26.c:1006:ll_write_end()) pos 6836224, len 4096, copied 4096 00000080:00200000:3.0:1734016138.688521:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a9d3fd80[3 ffff8800b3a5cdc8 2 1 ffff8800b5a11978] 00000080:00200000:3.0:1734016138.688524:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea00049b6480 2fffff00000805 3:0 ffff8800a9d3fd80 1669 lru 00000080:00200000:3.0:1734016138.688528:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a9d3fde0 645: 1< 0 - - > 2< 2641920 0 0 0x0 0x0 | ffff8800aa960708 ffff8800a90061a0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1734016138.688529:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a9d3fd80 00000080:00200000:3.0:1734016138.688530:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:3.0:1734016138.688531:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 1670 of 0 to 2236 bytes 00000080:00200000:3.0:1734016138.688547:0:10406:0:(rw26.c:1006:ll_write_end()) pos 6840320, len 2236, copied 2236 00000080:00200000:3.0:1734016138.688549:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a9d3fe58[3 ffff8800b3a5cdc8 2 1 ffff8800b5a11978] 00000080:00200000:3.0:1734016138.688551:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0004990900 2fffff00000805 3:0 ffff8800a9d3fe58 1670 lru 00000080:00200000:3.0:1734016138.688555:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a9d3feb8 646: 1< 0 - - > 2< 2646016 0 0 0x0 0x0 | ffff8800aa960708 ffff8800a90061a0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:3.0:1734016138.688557:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a9d3fe58 00000080:00200000:3.0:1734016138.688558:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 3. 00000080:00200000:3.0:1734016138.688562:0:10406:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 3, from 2837, to 2236 00000008:00010000:3.0:1734016138.688585:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8801369d4c00 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.688606:0:10406:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8801369d4c00 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.688614:0:10406:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8801369d4c00 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016138.688633:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134248, count 3, dirtied 3 00000080:00200000:3.0:1734016138.688640:0:10406:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 3 pages 7591 bytes, tot: 7591 00000080:00200000:3.0:1734016138.688645:0:10406:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 7591 00000020:00010000:3.0:1734016138.688659:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b39b0f38 (ffff88012bf7f1b8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016138.688661:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b39b0f38 (ffff88012bf7f1b8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.688663:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b58bab30 (ffff88012bf7f008/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016138.688668:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.688676:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.688694:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b39b0f38 (ffff88012bf7f1b8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.688697:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b58bab30 (ffff88012bf7f1b8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016138.688704:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 7591 bytes 00000080:00200000:3.0:1734016138.688711:0:10406:0:(file.c:1930:ll_file_io_generic()) Range unlock [1668, 1670] 00000080:00200000:3.0:1734016138.688717:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016138.688720:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 7591, restart: 0 00000080:00200000:3.0:1734016138.688725:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6842556 i_blocks 168 00000080:00200000:3.0:1734016138.688729:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 7591 00000080:00200008:3.0:1734016138.688745:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016138.688754:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016138.688778:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016138.688793:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6a5b lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116d08 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.688801:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6a5b lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116d08 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016138.688818:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:3.0:1734016138.688822:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e6a5b 00000080:00010000:3.0:1734016138.688829:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6a5b from it ffff8801369fbcd8 00010000:00010000:3.0:1734016138.688832:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6a5b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116d08 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.688837:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6a5b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116d08 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016138.688854:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016138.688860:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016138.688868:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.688870:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d44b0, osc lock ffff8800a8a277c0, flags 0x400001002 00000008:00010000:3.0:1734016138.688875:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.688877:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d44f0, osc lock ffff8800a8a278f0, flags 0x400001002 00010000:00010000:3.0:1734016138.688902:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.688910:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.688982:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016138.689006:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333907840:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.689137:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.689143:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.689190:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333907968:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.701577:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.701598:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.701610:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.701616:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.701622:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.701637:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.701644:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.701649:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.701655:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.701666:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa349d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.701671:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.701681:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.701686:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.701692:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e6a7e lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.701740:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333907840:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.704427:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.704450:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.705737:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.705754:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.705763:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.705782:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2648252, setting rss=2648252; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.705912:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.705919:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.705928:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.705936:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3486c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.705942:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.705954:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.705960:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.705965:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e6a85 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.706003:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333907968:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.711730:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335380992 00000100:00100000:3.0:1734016138.711734:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335380992 00000100:00100000:3.0:1734016138.711765:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12385 00000100:00100000:3.0:1734016138.711774:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335380864:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.711792:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.711804:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.711834:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 629, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.711836:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2648252 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 96 00000020:00010000:3.0:1734016138.711839:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.711840:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2648252 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 96 00000100:00100000:3.0:1734016138.711891:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335380864:12345-192.168.201.116@tcp:106: Request processed in 116us (280us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.711899:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12385 00000100:00100000:3.0:1734016138.730809:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333908352:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016138.752242:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333908864:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.756539:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335381376 00000100:00100000:3.0:1734016138.756548:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335381376 00000100:00100000:3.0:1734016138.756599:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12389 00000100:00100000:3.0:1734016138.756607:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335381376:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.756622:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.756632:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.756665:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 629, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.756668:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 72 00000020:00010000:3.0:1734016138.756671:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016138.756672:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 72 00000100:00100000:3.0:1734016138.756722:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335381376:12345-192.168.201.116@tcp:106: Request processed in 115us (735us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.756728:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12389 00010000:00010000:3.0:1734016138.759520:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.759538:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.759551:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.759557:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.759562:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.759578:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.759587:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.759597:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.759603:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.759609:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.759614:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.759624:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.759629:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.759635:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6aaf lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.759680:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333908864:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.760217:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.760225:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.760232:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.760238:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.760242:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.760249:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2648252, setting rss=2648252; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.760265:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.760271:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.760276:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.760281:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.760284:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.760290:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.760293:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.760298:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6ab6 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.760316:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333908992:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.766209:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016138.766227:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335381632 00000100:00100000:3.0:1734016138.766236:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335381632 00000100:00100000:3.0:1734016138.766281:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12391 00000100:00100000:3.0:1734016138.766288:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335381632:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.766302:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.766313:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.766338:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 629, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.766340:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2648252 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 96 00000020:00010000:3.0:1734016138.766343:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.766344:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2648252 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 96 00000100:00100000:3.0:1734016138.766387:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335381632:12345-192.168.201.116@tcp:106: Request processed in 100us (956us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.766393:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12391 00010000:00010000:3.0:1734016138.769138:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769157:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769173:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769186:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769191:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.769206:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.769213:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769219:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769225:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769230:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.769234:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769244:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.769250:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769259:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6abd lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.769282:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333909120:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.769297:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769303:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769310:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769315:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769319:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.769324:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2648252, setting rss=2648252; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.769338:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769343:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769352:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769357:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.769359:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769364:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.769366:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.769370:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6ac4 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.769381:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333909248:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.774916:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.774936:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.774949:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.774954:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.774961:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.774977:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.774984:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.774989:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.774996:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775002:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.775006:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775016:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.775021:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775025:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6acb lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.775067:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333909376:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016138.775411:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775420:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775426:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775430:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775436:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.775443:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2648252, setting rss=2648252; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.775554:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775560:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775566:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775570:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.775573:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775578:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.775581:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.775585:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e6ad2 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.775599:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333909504:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.779866:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333909632:192.168.201.116@tcp:22:fsx.0 00000080:00200000:3.0:1734016138.780842:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016138.780847:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016138.780855:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016138.780896:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [160, 160] 00000008:00010000:3.0:1734016138.780904:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016138.780907:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58ba630, osc lock ffff8800a8a27db0, flags 0x0 00010000:00010000:3.0:1734016138.780913:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.780920:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (655360 659455) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.780927:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.780932:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.780943:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016138.780952:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6842556 i_blocks 168 00000100:00100000:3.0:1734016138.780999:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016138.781012:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333909760:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016138.786524:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333909760:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016138.812116:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.815001:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.819570:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335382400 00000100:00100000:3.0:1734016138.819589:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335382400 00000100:00100000:3.0:1734016138.819650:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335382528 00000100:00100000:3.0:1734016138.819654:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335382528 00000100:00100000:3.0:1734016138.819678:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12396 00000100:00100000:3.0:1734016138.819688:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335382400:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.819707:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.819719:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.819775:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 629, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.819777:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2648252 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 96 00000020:00010000:3.0:1734016138.819780:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.819781:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2648252 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 96 00000100:00100000:3.0:1734016138.819853:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335382400:12345-192.168.201.116@tcp:106: Request processed in 165us (7748us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.819860:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12396 00000100:00100000:3.0:1734016138.819876:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12397 00000100:00100000:3.0:1734016138.819879:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335382528:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.819886:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.819892:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.819904:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 629, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.819905:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 72 00000020:00010000:3.0:1734016138.819908:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016138.819909:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 72 00000100:00100000:3.0:1734016138.819923:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335382528:12345-192.168.201.116@tcp:106: Request processed in 43us (4933us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.819927:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12397 00000100:00100000:3.0:1734016138.831106:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.832604:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016138.837607:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.837616:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.837623:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.837628:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.837633:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016138.837640:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2648252, setting rss=2648252; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016138.837655:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.837660:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.837665:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.837669:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016138.837671:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.837676:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016138.837679:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.837683:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6afc lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.837710:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333910528:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016138.842015:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335382912 00000100:00100000:3.0:1734016138.842026:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335382912 00000100:00100000:3.0:1734016138.842124:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12399 00000100:00100000:3.0:1734016138.842133:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335382912:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016138.842152:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e6a54 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116cfa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.842701:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016138.842709:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335382912:12345-192.168.201.116@tcp:104: Request processed in 577us (817us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.842716:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12399 00000100:00100000:3.0:1734016138.847451:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335383040 00000100:00100000:3.0:1734016138.847462:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335383040 00000100:00100000:3.0:1734016138.847529:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12400 00000100:00100000:3.0:1734016138.847536:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6466:x1818247335383040:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016138.847552:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6b03 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3084288->3088383] (req 3084288->3088383) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116db7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.847562:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6b03 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3084288->3088383] (req 3084288->3088383) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116db7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.847571:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6b03 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3084288->3088383) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27116db7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.847586:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6b03 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3084288->3088383) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27116db7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.847593:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6b03 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3084288->3088383) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27116db7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.847603:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6b03 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3084288->3088383) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc27116db7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.847621:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa367180) 00000100:00100000:3.0:1734016138.847674:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6466:x1818247335383040:12345-192.168.201.116@tcp:105: Request processed in 139us (844us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.847680:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12400 00000008:00010000:3.0:1734016138.859152:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6b03 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3084288->3088383) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116db7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.859190:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3088384 3092479) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6b03 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3084288->3088383) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116db7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016138.859222:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982d80 - object 0x240000400:6 offset 3088384<>3092480 00000080:00200008:3.0:1734016138.880266:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016138.880275:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:3.0:1734016138.880303:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016138.880333:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.880351:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016138.880363:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:3.0:1734016138.880366:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e6a9a 00000080:00010000:3.0:1734016138.880374:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6a9a from it ffff8801369fbcd8 00010000:00010000:3.0:1734016138.880377:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.880385:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016138.881208:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016138.881223:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016138.881283:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.881287:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4ab0, osc lock ffff8800a8a27c80, flags 0x400001002 00000008:00010000:3.0:1734016138.881308:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.881309:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4af0, osc lock ffff8800a8a27db0, flags 0x400001002 00010000:00010000:3.0:1734016138.881334:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e6b0a lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.881342:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e6b0a lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.881862:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.881871:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.881883:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.881890:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.885719:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016138.889817:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6842556 i_blocks 184 00000020:00010000:3.0:1734016138.889826:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016138.889829:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.889833:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4ab0 (ffff88012bf7f488/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.889837:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4af0 (ffff88012bf7f368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.889840:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.889842:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4ab0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.889845:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4af0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016138.889856:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016138.889881:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016138.889887:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016138.889896:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016138.889911:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.889913:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4ab0, osc lock ffff8800a8a27db0, flags 0x400001002 00000008:00010000:3.0:1734016138.889918:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.889919:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4af0, osc lock ffff8800a8a27c80, flags 0x400001002 00010000:00010000:3.0:1734016138.889937:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa348900/0xa50fd972cb8e6b11 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.889944:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa348900/0xa50fd972cb8e6b11 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.889997:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.890005:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.890012:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.890018:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.892367:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016138.896906:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6842556 i_blocks 184 00000020:00010000:3.0:1734016138.896917:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016138.896920:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.896925:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4ab0 (ffff88012bf7f368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.896930:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4af0 (ffff88012bf7f488/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.896933:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.896935:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4ab0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.896940:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4af0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016138.896955:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016138.896970:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 6842556 00000080:00200008:3.0:1734016138.896996:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016138.897005:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:3.0:1734016138.897026:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016138.897046:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.897059:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016138.897072:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:3.0:1734016138.897076:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e6a9a 00000080:00010000:3.0:1734016138.897081:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6a9a from it ffff8801369fbcd8 00010000:00010000:3.0:1734016138.897084:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.897090:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016138.897114:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016138.897125:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016138.897146:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.897148:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4ab0, osc lock ffff8800a8a27c80, flags 0x400001002 00000008:00010000:3.0:1734016138.897153:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.897154:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4af0, osc lock ffff8800a8a27db0, flags 0x400001002 00010000:00010000:3.0:1734016138.897184:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34bcc0/0xa50fd972cb8e6b18 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.897191:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34bcc0/0xa50fd972cb8e6b18 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.897251:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.897262:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.897270:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.897276:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.900909:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016138.904983:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6842556 i_blocks 184 00000020:00010000:3.0:1734016138.904992:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016138.904995:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.905001:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4ab0 (ffff88012bf7f488/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.905004:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4af0 (ffff88012bf7f368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.905007:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.905009:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4ab0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.905013:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4af0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016138.905025:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016138.905050:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016138.905056:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016138.905065:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016138.905083:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.905086:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4ab0, osc lock ffff8800a8a27db0, flags 0x400001002 00000008:00010000:3.0:1734016138.905091:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016138.905092:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4af0, osc lock ffff8800a8a27c80, flags 0x400001002 00010000:00010000:3.0:1734016138.905112:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6b1f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.905119:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34ad00/0xa50fd972cb8e6b1f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.905194:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.905203:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.905212:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.905220:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016138.908007:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016138.912458:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6842556 i_blocks 184 00000020:00010000:3.0:1734016138.912482:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016138.912485:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.912490:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4ab0 (ffff88012bf7f368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.912493:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4af0 (ffff88012bf7f488/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016138.912496:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.912497:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4ab0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016138.912502:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4af0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016138.912515:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016138.912527:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 6842556 00000080:00200008:3.0:1734016138.912606:0:10406:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8) current size 6842556, valid attrs 2068, mode 8801, uid -1227447776, gid -30720, new size 454749, atime 1734016138.000000000, mtime 1734016138.000000000, ctime 1734016138.000000000 00010000:00010000:3.0:1734016138.912634:0:10406:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.912652:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.912688:0:10406:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016138.912693:0:10406:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1734016138.912695:0:10406:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6a9a lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27116d4e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000020:00100000:3.0:1734016138.912714:0:10406:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff88012a571000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1734016138.912726:0:10406:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a7980a80 x1818247333912320/t0(0) o36->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016138.912745:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333912320:192.168.201.116@tcp:36:fsx.0 00000100:00100000:3.0:1734016138.912796:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4a80 going to sleep for 11 seconds 00000100:00100000:3.0:1734016138.917372:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.929610:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.937220:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.996026:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.996124:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016138.996153:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335384064 00000100:00100000:3.0:1734016138.996160:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335384064 00000100:00100000:3.0:1734016138.996196:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335384192 00000100:00100000:3.0:1734016138.996198:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335384192 00000100:00100000:3.0:1734016138.996209:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12408 00000100:00100000:3.0:1734016138.996215:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335384064:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.996227:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6b26 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116de1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.996238:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6b26 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116de1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.996259:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.996261:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 454749 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 56 00000020:00010000:3.0:1734016138.996264:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016138.996265:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 454749 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 56 00000100:00100000:3.0:1734016138.996300:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335384064:12345-192.168.201.116@tcp:106: Request processed in 85us (284us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.996306:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12408 00000100:00100000:3.0:1734016138.996316:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12409 00000100:00100000:3.0:1734016138.996319:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335384192:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016138.996325:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016138.996330:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016138.996339:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016138.996340:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 16 00000020:00010000:3.0:1734016138.996343:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016138.996344:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016138 atime: 1734016138 ctime: 1734016138 blocks: 16 00000100:00100000:3.0:1734016138.996354:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335384192:12345-192.168.201.116@tcp:106: Request processed in 34us (232us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016138.996359:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12409 00000080:00200000:3.0:1734016139.006164:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 454749 i_blocks 72 00000020:00010000:3.0:1734016139.006180:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_release():233 00000020:00010000:3.0:1734016139.006183:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.006188:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7e70 (ffff88012dc12998/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.006191:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7eb0 (ffff8800a8a543f8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.006192:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.006194:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7e70 (ffff88012dc12128/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.006197:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7eb0 (ffff88012dc12128/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016139.006209:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016139.006229:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016139.006235:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016139.006243:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016139.006257:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.006260:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4db0, osc lock ffff8800a8a27c80, flags 0x400001002 00000008:00010000:3.0:1734016139.006267:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.006269:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4df0, osc lock ffff8800a8a27db0, flags 0x400001002 00010000:00010000:3.0:1734016139.006288:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6b49 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.006295:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e6b49 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.006344:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333913728:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016139.006403:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.006410:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.006432:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016139.006438:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333913856:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.014247:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.026300:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016139.026519:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.026534:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.026547:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.026553:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.026559:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016139.026575:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016139.026584:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.026591:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.026597:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.026603:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa34a880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016139.026609:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.026619:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016139.026623:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.026628:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e6b50 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.026658:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333913856:192.168.201.116@tcp:101:fsx.0 00000080:00200000:3.0:1734016139.036959:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 454749 i_blocks 72 00000020:00010000:3.0:1734016139.036965:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_release():233 00000020:00010000:3.0:1734016139.036967:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.036972:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4db0 (ffff8800a8a543f8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.036975:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4df0 (ffff88012dc12998/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.036977:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.036979:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4db0 (ffff88012dc12128/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.036982:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4df0 (ffff88012dc12128/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016139.036993:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016139.037004:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 454749 00000080:00200008:3.0:1734016139.039441:0:10406:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8) current size 454749, valid attrs 2068, mode 8801, uid -1227447776, gid -30720, new size 8494918, atime 1734016139.000000000, mtime 1734016139.000000000, ctime 1734016139.000000000 00010000:00010000:3.0:1734016139.039507:0:10406:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6b2d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27116def expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016139.039533:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016139.039579:0:10406:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6b2d lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27116def expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016139.039584:0:10406:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1734016139.039587:0:10406:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e6b2d lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27116def expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000020:00100000:3.0:1734016139.039633:0:10406:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff88012a571000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1734016139.039644:0:10406:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a7982a00 x1818247333914112/t0(0) o36->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016139.039668:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333914112:192.168.201.116@tcp:36:fsx.0 00000100:00100000:3.0:1734016139.039737:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4d80 going to sleep for 11 seconds 00000100:00100000:3.0:1734016139.046515:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.053907:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333914112:192.168.201.116@tcp:36:fsx.0 00000080:00200000:3.0:1734016139.053948:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016139.053953:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016139.053973:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016139.053999:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1734016139.054014:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016139.054018:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4db0, osc lock ffff8800a8a27db0, flags 0x0 00000008:00010000:3.0:1734016139.054020:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016139.054021:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4df0, osc lock ffff8800a8a27c80, flags 0x0 00010000:00010000:3.0:1734016139.054039:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6b26 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116de1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.054051:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6b26 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116de1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.054061:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6b26 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116de1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.054068:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6b26 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116de1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.054073:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e6b26 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116de1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.054084:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.054090:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.054097:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.054106:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.054111:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.086045:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333914496:192.168.201.116@tcp:10:fsx.0 00000100:00100000:3.0:1734016139.086072:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333914368:192.168.201.116@tcp:10:fsx.0 00000100:00100000:3.0:1734016139.118496:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335384832 00000100:00100000:3.0:1734016139.118507:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335384832 00000100:00100000:3.0:1734016139.118734:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12414 00000100:00100000:3.0:1734016139.118742:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335384832:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.118754:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.118764:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.118786:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.118787:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016138 ctime: 1734016139 blocks: 16 00000020:00010000:3.0:1734016139.118790:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016139.118791:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016138 ctime: 1734016139 blocks: 16 00000100:00100000:3.0:1734016139.118830:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335384832:12345-192.168.201.116@tcp:106: Request processed in 89us (398us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.118838:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12414 00000100:00100000:3.0:1734016139.124290:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.124410:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.153823:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.153876:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016139.162278:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8494918 i_blocks 72 00000020:00010000:3.0:1734016139.162285:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef230 (ffff88012dc12998/0) at cl_lock_release():233 00000020:00010000:3.0:1734016139.162288:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef230 (ffff88012dc12998/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.162293:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7f30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.162297:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7f70 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.162299:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef230 (ffff88012dc12998/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.162302:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7f30 (ffff88012dc12998/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.162306:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7f70 (ffff88012dc12998/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016139.162317:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016139.162341:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016139.162346:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016139.162356:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016139.162376:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.162378:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4c30, osc lock ffff8800a8a278f0, flags 0x400001002 00000008:00010000:3.0:1734016139.162385:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.162386:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4c70, osc lock ffff8800a8a277c0, flags 0x400001002 00010000:00010000:3.0:1734016139.162408:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349440/0xa50fd972cb8e6b8f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.162416:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349440/0xa50fd972cb8e6b8f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.162502:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a1c0/0xa50fd972cb8e6b96 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.162509:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a1c0/0xa50fd972cb8e6b96 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.167505:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.167624:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016139.177580:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8494918 i_blocks 72 00000020:00010000:3.0:1734016139.177589:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef230 (ffff88012dc12998/0) at cl_lock_release():233 00000020:00010000:3.0:1734016139.177591:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef230 (ffff88012dc12998/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.177597:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4c30 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.177600:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4c70 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.177602:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef230 (ffff88012dc12998/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.177604:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4c30 (ffff88012dc12998/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.177607:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4c70 (ffff88012dc12998/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016139.177621:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016139.177635:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8494918 00000080:00200008:3.0:1734016139.177693:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 2947726, type: SEEK_SET 00000080:00200008:3.0:1734016139.177696:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 2947726, type: SEEK_SET, rc = 2947726 00000080:00200008:3.0:1734016139.177706:0:10406:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 2947726, count: 7266 00000080:00200000:3.0:1734016139.177712:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1734016139.177723:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016139.177765:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 719 00000080:00200000:3.0:1734016139.177802:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 2947726, bytes: 7266 00000080:00200000:3.0:1734016139.177806:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1734016139.177812:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:3.0:1734016139.177814:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [2947726, 2954992) 0 0 00000080:00200000:3.0:1734016139.177816:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016139.177828:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [2947726, 2954992) 2954992, 7266 00020000:00200000:3.0:1734016139.177833:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016139.177850:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [719, 721] 00000008:00010000:3.0:1734016139.177867:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016139.177870:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58ba530, osc lock ffff8800a8a277c0, flags 0x0 00010000:00010000:3.0:1734016139.177894:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6b9d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.177901:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6b9d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2945024->2957311] (req 2945024->2957311) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.177931:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6b9d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2945024->2957311] (req 2945024->2957311) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.177949:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333916288:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.179988:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4900 going to sleep for 11 seconds 00000100:00100000:3.0:1734016139.186164:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.193448:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333916288:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016139.193516:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6b9d lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2945024->2957311] (req 2945024->2957311) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116e7b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.193528:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6b9d lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2945024->2957311] (req 2945024->2957311) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116e7b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.193535:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6b9d lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2945024->2957311] (req 2945024->2957311) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116e7b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.208725:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016139.349910:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6ba4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.349939:0:1983:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6ba4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116e82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.349953:0:1983:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6ba4 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27116e82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.349965:0:1983:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6ba4 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116e82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016139.349980:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4300614, setting rss=4300614; kms=4300614, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6ba4 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116e82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.350005:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6ba4 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27116e82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.350012:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6ba4 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27116e82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.350017:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6ba4 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27116e82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.350023:0:1983:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e6ba4 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27116e82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.350042:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333916928:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.402006:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333917696:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.406953:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016139.406975:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335386368 00000100:00100000:3.0:1734016139.406986:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335386368 00000100:00100000:3.0:1734016139.407046:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12425 00000100:00100000:3.0:1734016139.407054:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335386368:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.407073:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.407084:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.407113:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.407116:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016138 ctime: 1734016139 blocks: 24 00000020:00010000:3.0:1734016139.407119:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016139.407120:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016138 ctime: 1734016139 blocks: 24 00000100:00100000:3.0:1734016139.407174:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335386368:12345-192.168.201.116@tcp:106: Request processed in 113us (515us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.407181:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12425 00010000:00010000:3.0:1734016139.412638:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.412660:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.412674:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.412681:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.412686:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016139.412701:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016139.412720:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.412725:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.412730:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.412735:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016139.412740:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.412750:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016139.412756:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.412761:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6bb2 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.415786:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333917696:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.424207:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335386496 00000100:00100000:3.0:1734016139.424218:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335386496 00000100:00100000:3.0:1734016139.424841:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12426 00000100:00100000:3.0:1734016139.424850:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335386496:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.424867:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.424879:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.424907:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.424909:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016138 ctime: 1734016139 blocks: 24 00000020:00010000:3.0:1734016139.424912:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016139.424913:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016138 ctime: 1734016139 blocks: 24 00000100:00100000:3.0:1734016139.424968:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335386496:12345-192.168.201.116@tcp:106: Request processed in 118us (1432us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.424974:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12426 00010000:00010000:3.0:1734016139.427402:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.427423:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.427435:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.427442:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.427447:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016139.427461:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016139.427509:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.427515:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.427521:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.427527:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016139.427531:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.427541:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016139.427546:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.427552:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6bb9 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.427588:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333917824:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.434938:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335386624 00000100:00100000:3.0:1734016139.434951:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335386624 00000100:00100000:3.0:1734016139.435008:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12427 00000100:00100000:3.0:1734016139.435017:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335386624:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016139.435046:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e6bab lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1511424->1519615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116e89 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.435145:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016139.435151:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335386624:12345-192.168.201.116@tcp:104: Request processed in 136us (320us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.435160:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12427 00000100:00100000:3.0:1734016139.440940:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335386752 00000100:00100000:3.0:1734016139.440950:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335386752 00000100:00100000:3.0:1734016139.440994:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12428 00000100:00100000:3.0:1734016139.441001:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335386752:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016139.441017:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4407296->4415487] (req 4407296->4415487) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.441028:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4407296->4415487] (req 4407296->4415487) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.441037:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.441052:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.441060:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.441070:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.441090:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa365d40) 00000100:00100000:3.0:1734016139.441148:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335386752:12345-192.168.201.116@tcp:105: Request processed in 147us (295us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.441154:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12428 00010000:00010000:3.0:1734016139.465862:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.465878:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.465952:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016139.465972:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333918592:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016139.474627:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.474648:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.474661:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.474667:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.474674:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016139.474689:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016139.474697:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.474702:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.474709:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.474713:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa348fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016139.474718:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.474729:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016139.474733:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.474743:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6bdc lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.474792:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333918592:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.483895:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335387392 00000100:00100000:3.0:1734016139.483906:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335387392 00000100:00100000:3.0:1734016139.483943:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12433 00000100:00100000:3.0:1734016139.483950:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335387392:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016139.483964:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e6a31 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3346432->3350527) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116cd0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.484021:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016139.484027:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335387392:12345-192.168.201.116@tcp:104: Request processed in 77us (3181us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.484032:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12433 00000100:00100000:3.0:1734016139.500242:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335387648 00000100:00100000:3.0:1734016139.500249:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335387648 00000100:00100000:3.0:1734016139.500275:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12435 00000100:00100000:3.0:1734016139.500281:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335387648:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.500292:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.500301:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.500325:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.500327:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4414343 mtime: 1734016139 atime: 1734016138 ctime: 1734016139 blocks: 64 00000020:00010000:3.0:1734016139.500331:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016139.500332:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4414343 mtime: 1734016139 atime: 1734016138 ctime: 1734016139 blocks: 64 00000100:00100000:3.0:1734016139.500359:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335387648:12345-192.168.201.116@tcp:106: Request processed in 77us (194us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.500365:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12435 00000100:00100000:3.0:1734016139.518802:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335387776 00000100:00100000:3.0:1734016139.518812:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335387776 00000100:00100000:3.0:1734016139.518853:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12436 00000100:00100000:3.0:1734016139.518860:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335387776:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.518876:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.518888:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.518916:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.518918:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4414343 mtime: 1734016139 atime: 1734016138 ctime: 1734016139 blocks: 64 00000020:00010000:3.0:1734016139.518922:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016139.518923:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4414343 mtime: 1734016139 atime: 1734016138 ctime: 1734016139 blocks: 64 00000100:00100000:3.0:1734016139.518970:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335387776:12345-192.168.201.116@tcp:106: Request processed in 111us (309us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.518976:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12436 00000100:00100000:3.0:1734016139.525430:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335387904 00000100:00100000:3.0:1734016139.525441:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335387904 00000100:00100000:3.0:1734016139.525665:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12437 00000100:00100000:3.0:1734016139.525672:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335387904:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.525686:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e6be3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2404352->2412543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ec1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.525695:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e6be3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2404352->2412543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ec1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.525722:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.525725:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 24 00000020:00010000:3.0:1734016139.525727:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016139.525728:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 24 00000100:00100000:3.0:1734016139.525774:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335387904:12345-192.168.201.116@tcp:106: Request processed in 102us (510us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.525780:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12437 00000100:00100000:3.0:1734016139.548677:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016139.548697:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335388160 00000100:00100000:3.0:1734016139.548707:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335388160 00000100:00100000:3.0:1734016139.548749:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12439 00000100:00100000:3.0:1734016139.548757:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3c450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335388160:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.548771:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.548781:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.548813:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.548816:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4414343 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 64 00000020:00010000:3.0:1734016139.548818:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016139.548819:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4414343 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 64 00000100:00100000:3.0:1734016139.548870:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3c450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335388160:12345-192.168.201.116@tcp:106: Request processed in 114us (343us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.548876:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12439 00010000:00010000:3.0:1734016139.560047:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.560069:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.560084:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.560090:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.560096:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016139.560110:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4414343, setting rss=4414343; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016139.560126:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.560136:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.560141:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.560146:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367cc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016139.560151:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.560171:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016139.560177:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.560182:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e6c06 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.563148:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333919616:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.569054:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335388288 00000100:00100000:3.0:1734016139.569068:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335388288 00000100:00100000:3.0:1734016139.569127:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12440 00000100:00100000:3.0:1734016139.569134:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335388288:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016139.569151:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e6bc0 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4407296->4415487) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116e9e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.569255:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016139.569260:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335388288:12345-192.168.201.116@tcp:104: Request processed in 127us (3395us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.569266:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12440 00000100:00100000:3.0:1734016139.579004:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335388416 00000100:00100000:3.0:1734016139.579013:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335388416 00000100:00100000:3.0:1734016139.579056:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12441 00000100:00100000:3.0:1734016139.579062:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335388416:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016139.579076:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6c0d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5263360->5275647] (req 5263360->5275647) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116eeb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.579086:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6c0d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5263360->5275647] (req 5263360->5275647) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116eeb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.579094:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6c0d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5263360->5275647) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116eeb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.579103:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6c0d lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5263360->5275647) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116eeb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.579109:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6c0d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5263360->5275647) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116eeb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.579118:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6c0d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5263360->5275647) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116eeb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.579310:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa364b40) 00000100:00100000:3.0:1734016139.579363:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335388416:12345-192.168.201.116@tcp:105: Request processed in 301us (558us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.579370:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12441 00000100:00100000:3.0:1734016139.590333:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335388544 00000100:00100000:3.0:1734016139.590345:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335388544 00000100:00100000:3.0:1734016139.590399:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12442 00000100:00100000:3.0:1734016139.590407:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335388544:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.590421:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e6be3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2404352->2412543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ec1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.590432:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e6be3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2404352->2412543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116ec1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.590460:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.590462:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 24 00000020:00010000:3.0:1734016139.590478:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016139.590479:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 24 00000100:00100000:3.0:1734016139.590535:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335388544:12345-192.168.201.116@tcp:106: Request processed in 128us (1505us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.590542:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12442 00010000:00010000:3.0:1734016139.595309:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.595326:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.595339:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.595345:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.595350:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016139.595362:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5271818, setting rss=5271818; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016139.595369:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.595373:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.595379:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.595385:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016139.595389:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.595398:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016139.595402:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.595412:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6c14 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.597046:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333920256:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016139.597089:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.597100:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.597111:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.597116:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.597122:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016139.597132:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016139.597300:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.597305:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.597316:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.597322:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016139.597326:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.597334:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016139.597337:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.597342:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6c1b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.597358:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333920384:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.602978:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12445 00000100:00100000:3.0:1734016139.602984:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335388928:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.602991:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6c0d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5263360->5275647) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116eeb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.602997:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6c0d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5263360->5275647) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116eeb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.603008:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.603009:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5271818 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 104 00000020:00010000:3.0:1734016139.603011:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016139.603012:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5271818 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 104 00000100:00100000:3.0:1734016139.603637:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335388928:12345-192.168.201.116@tcp:106: Request processed in 653us (869us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.603641:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12445 00000080:00200000:3.0:1734016139.607452:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9466122 i_blocks 128 00000020:00010000:3.0:1734016139.607461:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016139.607498:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.607504:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7cf0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.607508:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7d30 (ffff8800a8a54368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.607509:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.607511:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7cf0 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.607515:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7d30 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016139.607527:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016139.607538:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9466122 00000080:00200008:3.0:1734016139.607557:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016139.607563:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:3.0:1734016139.607581:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016139.607595:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016139.607604:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016139.607615:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:3.0:1734016139.607617:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e6b57 00000080:00010000:3.0:1734016139.607622:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6b57 from it ffff8801369fbcd8 00010000:00010000:3.0:1734016139.607625:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016139.607631:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016139.607651:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016139.607660:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016139.607673:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.607675:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4e70, osc lock ffff8800a8a278f0, flags 0x400001002 00000008:00010000:3.0:1734016139.607679:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.607680:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4eb0, osc lock ffff8800a8a27b50, flags 0x400001002 00010000:00010000:3.0:1734016139.607696:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a640/0xa50fd972cb8e6c30 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.607703:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a640/0xa50fd972cb8e6c30 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.607755:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348d80/0xa50fd972cb8e6c37 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.607761:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa348d80/0xa50fd972cb8e6c37 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.613863:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.613936:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016139.618986:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9466122 i_blocks 128 00000020:00010000:3.0:1734016139.618993:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016139.618996:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.619001:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4e70 (ffff8800a8a54368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.619005:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4eb0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.619007:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.619008:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4e70 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.619012:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4eb0 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016139.619024:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016139.619049:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016139.619053:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016139.619065:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016139.619076:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [772, 773] 00000008:00010000:3.0:1734016139.619087:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016139.619090:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58bacb0, osc lock ffff8800a8a27b50, flags 0x0 00010000:00010000:3.0:1734016139.619108:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.619114:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3162112->3170303] (req 3162112->3170303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.619144:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3162112->3170303] (req 3162112->3170303) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.619177:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333921024:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.619231:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d49c0 going to sleep for 11 seconds 00000100:00100000:3.0:1734016139.621869:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.627703:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335389440 00000100:00100000:3.0:1734016139.627711:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335389440 00000100:00100000:3.0:1734016139.627783:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12448 00000100:00100000:3.0:1734016139.627790:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335389440:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016139.627800:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3162112->3170303] (req 3162112->3170303) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116f1c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.627814:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3162112->3170303] (req 3162112->3170303) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116f1c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.627822:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3162112->3170303) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116f1c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.627830:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3162112->3170303) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116f1c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.627838:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3162112->3170303) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116f1c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.627846:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3162112->3170303) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116f1c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.627875:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa34b600) 00000100:00100000:3.0:1734016139.627924:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335389440:12345-192.168.201.116@tcp:105: Request processed in 134us (473us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.627930:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12448 00000100:00100000:3.0:1734016139.637671:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.648193:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.665781:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.674745:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.685109:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.703322:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.711793:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.716349:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335390464 00000100:00100000:3.0:1734016139.716358:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335390464 00000100:00100000:3.0:1734016139.716392:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12456 00000100:00100000:3.0:1734016139.716399:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335390464:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016139.716412:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e6c6f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [811008->819199] (req 811008->819199) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116f4d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.716423:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e6c6f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [811008->819199] (req 811008->819199) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27116f4d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.716432:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e6c6f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 811008->819199) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27116f4d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.716440:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e6c6f lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 811008->819199) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27116f4d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.716448:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e6c6f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 811008->819199) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27116f4d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.716456:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e6c6f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 811008->819199) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc27116f4d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.716506:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c06c0) 00000100:00100000:3.0:1734016139.716550:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335390464:12345-192.168.201.116@tcp:105: Request processed in 151us (271us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.716556:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12456 00000100:00100000:3.0:1734016139.856021:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335390592 00000100:00100000:3.0:1734016139.856030:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335390592 00000100:00100000:3.0:1734016139.856361:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12457 00000100:00100000:3.0:1734016139.856368:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335390592:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.856380:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3162112->3170303) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f1c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.856389:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3162112->3170303) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f1c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.856411:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 2313, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.856413:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5271818 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 128 00000020:00010000:3.0:1734016139.856416:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 4, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016139.856417:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5271818 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 128 00000100:00100000:3.0:1734016139.856653:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335390592:12345-192.168.201.116@tcp:106: Request processed in 283us (1310us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.856659:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12457 00000100:00100000:3.0:1734016139.881647:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335390720 00000100:00100000:3.0:1734016139.881655:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335390720 00000100:00100000:3.0:1734016139.882117:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12458 00000100:00100000:3.0:1734016139.882124:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335390720:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016139.882136:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3162112->3170303) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f1c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.882145:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6c3e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3162112->3170303) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f1c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.882176:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 2313, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016139.882178:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5271818 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 128 00000020:00010000:3.0:1734016139.882182:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016139.882183:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5271818 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 128 00000100:00100000:3.0:1734016139.882525:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335390720:12345-192.168.201.116@tcp:106: Request processed in 400us (1821us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016139.882531:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12458 00010000:00010000:3.0:1734016139.886126:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.886141:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.886151:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.886168:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.886173:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016139.886185:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5271818, setting rss=5271818; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016139.886196:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.886201:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.886211:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.886217:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016139.886221:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.886230:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016139.886236:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.886241:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6c84 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.887155:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333923456:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.902270:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333923584:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016139.902307:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1990656->1998847] (req 1990656->1998847) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.902320:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1990656->1998847] (req 1990656->1998847) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.902328:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1990656->1998847] (req 1990656->1998847) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.905092:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000020:00010000:3.0:1734016139.912866:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32c68 (ffff8800a8a546c8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016139.912871:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32c68 (ffff8800a8a546c8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.912876:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811830 (ffff88012dc12008/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016139.912885:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.912896:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016139.912910:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32c68 (ffff8800a8a546c8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.912912:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811830 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016139.912925:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016139.912946:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016139.912951:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016139.912966:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016139.912978:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6b5e lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e3c expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016139.912987:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6b5e lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e3c expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016139.912995:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:3.0:1734016139.912997:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e6b5e 00000080:00010000:3.0:1734016139.913001:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6b5e from it ffff8801369fbcd8 00010000:00010000:3.0:1734016139.913004:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6b5e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116e3c expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016139.913009:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6b5e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e3c expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016139.913023:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016139.913035:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016139.913053:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.913058:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d47b0, osc lock ffff8800a8a278f0, flags 0x400001002 00000008:00010000:3.0:1734016139.913062:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.913063:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d47f0, osc lock ffff8800a8a27a20, flags 0x400001002 00010000:00010000:3.0:1734016139.913082:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34aac0/0xa50fd972cb8e6c92 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.913090:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34aac0/0xa50fd972cb8e6c92 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.913126:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.913133:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.913141:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.913146:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.917747:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016139.920350:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9466122 i_blocks 152 00000020:00010000:3.0:1734016139.920357:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016139.920360:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.920366:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d47b0 (ffff88012dc12008/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.920369:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d47f0 (ffff8800a8a54368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.920372:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.920375:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d47b0 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.920379:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d47f0 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016139.920389:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016139.920409:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016139.920414:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016139.920423:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016139.920437:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.920439:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d47b0, osc lock ffff8800a8a27a20, flags 0x400001002 00000008:00010000:3.0:1734016139.920444:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.920445:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d47f0, osc lock ffff8800a8a278f0, flags 0x400001002 00010000:00010000:3.0:1734016139.920498:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3498c0/0xa50fd972cb8e6c99 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.920506:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3498c0/0xa50fd972cb8e6c99 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.920546:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.920554:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.920562:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.920568:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.923853:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016139.932861:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9466122 i_blocks 152 00000020:00010000:3.0:1734016139.932869:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016139.932871:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.932876:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d47b0 (ffff8800a8a54368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.932880:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d47f0 (ffff88012dc12008/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.932882:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.932883:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d47b0 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.932886:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d47f0 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016139.932897:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016139.932908:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9466122 00000080:00a00000:3.0:1734016139.932969:0:10406:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800b61fa288 start=0x7f17cf5cb000 end=0x7f17cf5cc000 vm_flags=0xfb 00000080:00200000:3.0:1734016139.932978:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016139.932988:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016139.933008:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.933010:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d47b0, osc lock ffff8800a8a278f0, flags 0x400001002 00000008:00010000:3.0:1734016139.933015:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016139.933016:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d47f0, osc lock ffff8800a8a27a20, flags 0x400001002 00010000:00010000:3.0:1734016139.933027:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800a9dd4000 type 11 mode 6 res 9663677440/6 (0 18446744073709551615) 00010000:00010000:3.0:1734016139.933038:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34ba80/0xa50fd972cb8e6ca0 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 4 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.933045:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34ba80/0xa50fd972cb8e6ca0 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.933091:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.933098:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.933104:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.933109:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.938698:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016139.942298:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9466122 i_blocks 152 00000020:00010000:3.0:1734016139.942305:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016139.942308:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.942313:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d47b0 (ffff88012dc12008/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.942317:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d47f0 (ffff8800a8a54368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016139.942320:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.942322:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d47b0 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016139.942325:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d47f0 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016139.942337:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016139.942410:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 594 00000080:00200000:3.0:1734016139.942420:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016139.942438:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016139.942449:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [594, 594] 00000008:00010000:3.0:1734016139.942461:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016139.942497:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58ba9b0, osc lock ffff8800a8a27a20, flags 0x0 00010000:00010000:3.0:1734016139.942517:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34b840/0xa50fd972cb8e6ca7 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.942524:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34b840/0xa50fd972cb8e6ca7 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2433024->2437119] (req 2433024->2437119) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016139.942548:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34b840/0xa50fd972cb8e6ca7 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2433024->2437119] (req 2433024->2437119) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016139.942570:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333924480:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016139.942614:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4480 going to sleep for 11 seconds 00000100:00100000:3.0:1734016139.945401:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016139.966385:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.004671:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333925376:192.168.201.116@tcp:16:fsx.0 00000100:00100000:3.0:1734016140.047115:0:1983:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0000-osc-ffff88012a5ff800: committing for last_committed 4294968056 gen 2 00000100:00100000:3.0:1734016140.047176:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333925376:192.168.201.116@tcp:16:fsx.0 00000100:00100000:3.0:1734016140.060237:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335391872 00000100:00100000:3.0:1734016140.060245:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335391872 00000100:00100000:3.0:1734016140.060277:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335392000 00000100:00100000:3.0:1734016140.060278:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335392000 00000100:00100000:3.0:1734016140.060291:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12467 00000100:00100000:3.0:1734016140.060297:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335391872:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.060311:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.060321:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.060344:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1026, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.060346:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 24 00000020:00010000:3.0:1734016140.060349:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016140.060350:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 24 00000100:00100000:3.0:1734016140.060393:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335391872:12345-192.168.201.116@tcp:106: Request processed in 94us (395us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.060400:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12467 00000100:00100000:3.0:1734016140.060414:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12468 00000100:00100000:3.0:1734016140.060417:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335392000:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.060423:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34b840/0xa50fd972cb8e6ca7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2433024->2437119) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f85 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.060429:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34b840/0xa50fd972cb8e6ca7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2433024->2437119) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116f85 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.060438:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1026, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.060440:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5271818 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 136 00000020:00010000:3.0:1734016140.060442:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016140.060443:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5271818 mtime: 1734016139 atime: 1734016139 ctime: 1734016139 blocks: 136 00000100:00100000:3.0:1734016140.060458:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335392000:12345-192.168.201.116@tcp:106: Request processed in 40us (359us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.060462:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12468 00000080:00200000:3.0:1734016140.063747:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9466122 i_blocks 160 00000020:00010000:3.0:1734016140.063754:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016140.063756:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016140.063761:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee76f0 (ffff8800a8a54908/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016140.063764:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7730 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016140.063766:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016140.063768:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee76f0 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016140.063772:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7730 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016140.063781:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016140.063799:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016140.063803:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016140.063811:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016140.063823:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016140.063825:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4cf0, osc lock ffff8800a8a278f0, flags 0x400001002 00000008:00010000:3.0:1734016140.063829:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016140.063830:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4d30, osc lock ffff8800a8a27430, flags 0x400001002 00010000:00010000:3.0:1734016140.063846:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa349b00/0xa50fd972cb8e6cbc lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.063853:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa349b00/0xa50fd972cb8e6cbc lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.063898:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333925760:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016140.063948:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349200/0xa50fd972cb8e6cc3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.063954:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349200/0xa50fd972cb8e6cc3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.063976:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016140.063981:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333925888:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.068588:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.068711:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016140.072736:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9466122 i_blocks 160 00000020:00010000:3.0:1734016140.072743:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016140.072745:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016140.072749:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4cf0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016140.072752:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4d30 (ffff8800a8a54908/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016140.072754:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cb6c0 (ffff8800a8a546c8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016140.072756:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4cf0 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016140.072759:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4d30 (ffff8800a8a546c8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016140.072769:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016140.072779:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9466122 00000080:00200008:3.0:1734016140.072835:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016140.072840:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:3.0:1734016140.072853:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016140.072864:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016140.072872:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016140.072879:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:3.0:1734016140.072882:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e6b57 00000080:00010000:3.0:1734016140.072886:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6b57 from it ffff8801369fbcd8 00010000:00010000:3.0:1734016140.072888:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016140.072893:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016140.072909:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016140.072918:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016140.072931:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016140.072932:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4cf0, osc lock ffff8800a8a27430, flags 0x400001002 00000008:00010000:3.0:1734016140.072936:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016140.072937:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4d30, osc lock ffff8800a8a278f0, flags 0x400001002 00010000:00010000:3.0:1734016140.072951:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa349f80/0xa50fd972cb8e6cca lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.072958:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa349f80/0xa50fd972cb8e6cca lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.072992:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.072997:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.076775:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.078619:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016140.083535:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.083552:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.083561:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.083566:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.083571:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.083583:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.083597:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.083603:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.083609:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.083615:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa34b600 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.083618:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.083630:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.083634:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.083639:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34b600/0xa50fd972cb8e6cd1 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.083671:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333926144:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.090332:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016140.090353:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335392640 00000100:00100000:3.0:1734016140.090360:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335392640 00000100:00100000:3.0:1734016140.090433:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12473 00000100:00100000:3.0:1734016140.090440:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3fc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335392640:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016140.090456:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34b840/0xa50fd972cb8e6ca7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2433024->2437119) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116f85 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.090788:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016140.090794:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3fc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335392640:12345-192.168.201.116@tcp:104: Request processed in 354us (780us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.090800:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12473 00000100:00100000:3.0:1734016140.097595:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335392768 00000100:00100000:3.0:1734016140.097602:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335392768 00000100:00100000:3.0:1734016140.097638:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12474 00000100:00100000:3.0:1734016140.097644:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6466:x1818247335392768:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016140.097655:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5529600->5537791] (req 5529600->5537791) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.097664:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5529600->5537791] (req 5529600->5537791) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.097672:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.097680:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.097687:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.097695:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.097715:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c1f80) 00000100:00100000:3.0:1734016140.097753:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6466:x1818247335392768:12345-192.168.201.116@tcp:105: Request processed in 109us (631us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.097759:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12474 00000100:00100000:3.0:1734016140.107212:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335392896 00000100:00100000:3.0:1734016140.107219:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335392896 00000100:00100000:3.0:1734016140.107250:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12475 00000100:00100000:3.0:1734016140.107256:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335392896:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.107268:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.107280:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.107299:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.107300:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016139 atime: 1734016139 ctime: 1734016140 blocks: 152 00000020:00010000:3.0:1734016140.107304:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016140.107304:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016139 atime: 1734016139 ctime: 1734016140 blocks: 152 00000100:00100000:3.0:1734016140.107337:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335392896:12345-192.168.201.116@tcp:106: Request processed in 82us (221us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.107343:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12475 00010000:00010000:3.0:1734016140.109720:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.109737:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.109748:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.109753:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.109758:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.109772:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5535168, setting rss=5535168; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.110041:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.110047:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.110052:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.110057:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.110061:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.110071:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.110075:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.110080:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6cdf lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.111245:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333926656:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.114100:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335393024 00000100:00100000:3.0:1734016140.114108:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335393024 00000100:00100000:3.0:1734016140.114142:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12476 00000100:00100000:3.0:1734016140.114149:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335393024:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.114169:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.114181:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.114204:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.114206:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016139 atime: 1734016139 ctime: 1734016140 blocks: 152 00000020:00010000:3.0:1734016140.114209:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016140.114210:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016139 atime: 1734016139 ctime: 1734016140 blocks: 152 00000100:00100000:3.0:1734016140.114250:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335393024:12345-192.168.201.116@tcp:106: Request processed in 100us (241us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.114256:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12476 00010000:00010000:3.0:1734016140.116882:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.116896:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.116908:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.116914:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.116919:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.116933:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5535168, setting rss=5535168; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.116944:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.116949:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.116955:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.116960:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.116965:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.116973:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.116978:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.116984:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365f80/0xa50fd972cb8e6ce6 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.117023:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333926784:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.122080:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335393152 00000100:00100000:3.0:1734016140.122089:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335393152 00000100:00100000:3.0:1734016140.122185:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12477 00000100:00100000:3.0:1734016140.122193:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335393152:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016140.122208:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6c8b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1990656->1998847) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116f69 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.122566:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016140.122573:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335393152:12345-192.168.201.116@tcp:104: Request processed in 381us (755us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.122580:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12477 00000100:00100000:3.0:1734016140.139004:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.170091:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.170193:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.193166:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.194443:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.208496:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335393920 00000100:00100000:3.0:1734016140.208510:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335393920 00000100:00100000:3.0:1734016140.208563:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12482 00000100:00100000:3.0:1734016140.208572:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335393920:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.208586:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.208597:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.208622:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 67, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.208624:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016140 atime: 1734016139 ctime: 1734016140 blocks: 24 00000020:00010000:3.0:1734016140.208627:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016140.208628:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016140 atime: 1734016139 ctime: 1734016140 blocks: 24 00000100:00100000:3.0:1734016140.208674:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335393920:12345-192.168.201.116@tcp:106: Request processed in 104us (1912us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.208680:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12482 00000100:00100000:3.0:1734016140.209786:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335394048 00000100:00100000:3.0:1734016140.209791:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335394048 00000100:00100000:3.0:1734016140.209822:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12483 00000100:00100000:3.0:1734016140.209827:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335394048:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.209836:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.209844:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.209860:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 67, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.209862:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016139 ctime: 1734016140 blocks: 152 00000020:00010000:3.0:1734016140.209864:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016140.209865:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016139 ctime: 1734016140 blocks: 152 00000100:00100000:3.0:1734016140.209892:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335394048:12345-192.168.201.116@tcp:106: Request processed in 65us (241us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.209897:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12483 00000100:00100000:3.0:1734016140.242722:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016140.242746:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335394432 00000100:00100000:3.0:1734016140.242756:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335394432 00000100:00100000:3.0:1734016140.242796:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12486 00000100:00100000:3.0:1734016140.242802:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3fc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335394432:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016140.242816:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e6cd8 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5529600->5537791) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27116fb6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.242964:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016140.242972:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3fc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335394432:12345-192.168.201.116@tcp:104: Request processed in 168us (378us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.242980:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12486 00000100:00100000:3.0:1734016140.248868:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335394560 00000100:00100000:3.0:1734016140.248876:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335394560 00000100:00100000:3.0:1734016140.248909:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12487 00000100:00100000:3.0:1734016140.248915:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335394560:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016140.248928:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d2c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2019328->2027519] (req 2019328->2027519) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711700a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.248936:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d2c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2019328->2027519] (req 2019328->2027519) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711700a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.248943:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d2c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2019328->2027519) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711700a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.248953:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d2c lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2019328->2027519) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711700a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.248961:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d2c lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2019328->2027519) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711700a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.248969:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d2c lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2019328->2027519) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711700a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.248991:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa367180) 00000100:00100000:3.0:1734016140.249034:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335394560:12345-192.168.201.116@tcp:105: Request processed in 118us (249us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.249039:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12487 00000100:00100000:3.0:1734016140.409678:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.423068:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333929728:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.426383:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.426409:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335394816 00000100:00100000:3.0:1734016140.426417:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335394816 00000100:00100000:3.0:1734016140.426457:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12489 00000100:00100000:3.0:1734016140.426481:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335394816:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.426494:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.426503:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.426526:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.426528:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016140 atime: 1734016139 ctime: 1734016140 blocks: 32 00000020:00010000:3.0:1734016140.426531:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016140.426532:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016140 atime: 1734016139 ctime: 1734016140 blocks: 32 00000100:00100000:3.0:1734016140.429573:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335394816:12345-192.168.201.116@tcp:106: Request processed in 3100us (3189us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.429584:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12489 00010000:00010000:3.0:1734016140.432512:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.432527:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.432544:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.432550:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.432555:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.432568:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.432574:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.432578:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.432583:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.432587:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.432593:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.432602:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.432605:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.432610:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e6d3a lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.436786:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333929728:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.443829:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333929856:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016140.443856:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3387392->3391487] (req 3387392->3391487) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.443865:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3387392->3391487] (req 3387392->3391487) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.443870:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3387392->3391487] (req 3387392->3391487) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.463246:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335395072 00000100:00100000:3.0:1734016140.463258:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335395072 00000100:00100000:3.0:1734016140.463308:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12491 00000100:00100000:3.0:1734016140.463315:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6466:x1818247335395072:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016140.463330:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3387392->3391487] (req 3387392->3391487) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463340:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3387392->3391487] (req 3387392->3391487) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463347:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463360:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463367:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463375:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463398:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa3646c0) 00000100:00100000:3.0:1734016140.463451:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6466:x1818247335395072:12345-192.168.201.116@tcp:105: Request processed in 136us (8789us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.463456:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12491 00010000:00010000:3.0:1734016140.463493:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463502:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463524:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.463534:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 5535168, setting rss=5535168; kms=5535168, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463544:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463555:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016140.463568:0:10406:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [3390342, 3391326) 00000080:00200000:3.0:1734016140.463584:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 827 of 2950 to 984 bytes 00000100:00100000:3.0:1734016140.463684:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:3.0:1734016140.463700:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8801369d4c00 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.463729:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3387392 3391487) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016140.463759:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982d80 - object 0x240000400:6 offset 3387392<>3391488 00000100:00100000:3.0:1734016140.463774:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333930240:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016140.474250:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.474344:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.475120:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016140.475130:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335395200 00000100:00100000:3.0:1734016140.475133:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335395200 00000100:00100000:3.0:1734016140.475166:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12492 00000100:00100000:3.0:1734016140.475171:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dc050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335395200:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.475182:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.475191:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.475216:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.475218:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016139 ctime: 1734016140 blocks: 152 00000020:00010000:3.0:1734016140.475220:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016140.475221:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016139 ctime: 1734016140 blocks: 152 00000100:00100000:3.0:1734016140.475263:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dc050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335395200:12345-192.168.201.116@tcp:106: Request processed in 90us (1014us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.475269:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12492 00000100:00100000:3.0:1734016140.497159:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.500648:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.500676:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335395584 00000100:00100000:3.0:1734016140.500684:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335395584 00000100:00100000:3.0:1734016140.500716:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12494 00000100:00100000:3.0:1734016140.500721:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335395584:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.500735:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.500745:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.500768:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.500769:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016139 ctime: 1734016140 blocks: 152 00000020:00010000:3.0:1734016140.500772:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016140.500773:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016139 ctime: 1734016140 blocks: 152 00000100:00100000:3.0:1734016140.500811:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335395584:12345-192.168.201.116@tcp:106: Request processed in 89us (173us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.500820:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12494 00000100:00100000:3.0:1734016140.508444:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.508775:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.508796:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335395712 00000100:00100000:3.0:1734016140.508803:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335395712 00000100:00100000:3.0:1734016140.509056:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335395840 00000100:00100000:3.0:1734016140.509059:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335395840 00010000:00010000:3.0:1734016140.512771:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512787:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512800:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512805:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512810:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.512822:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.512829:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512834:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512841:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512845:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.512849:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512859:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.512864:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512868:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6d6b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.512907:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333931008:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016140.512921:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512926:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512932:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512936:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512941:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.512946:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5535168, setting rss=5535168; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.512952:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512956:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512961:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512966:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.512968:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512973:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.512975:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.512979:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6d64 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.512990:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333930880:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016140.524979:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.524992:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.525003:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.525008:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.525013:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.525025:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.525032:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.525037:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.525043:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.525048:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.525052:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.525060:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.525064:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.525069:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6d79 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.525191:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333931264:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016140.526376:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.526384:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.526390:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.526394:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.526399:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.526407:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5535168, setting rss=5535168; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.526413:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.526418:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.526423:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.526427:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.526429:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.526434:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.526437:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.526442:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e6d72 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.526452:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333931136:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016140.532771:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.532796:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.532815:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333931520:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016140.540215:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7900 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.540258:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7900 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.540279:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333931776:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016140.541059:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.541076:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3358720 3362815) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 7/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016140.541101:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982d80 - object 0x280000400:8 offset 3358720<>3362816 00000008:00010000:3.0:1734016140.544193:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.544230:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.544250:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333932032:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016140.604089:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335396352 00000100:00100000:3.0:1734016140.604095:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335396352 00000100:00100000:3.0:1734016140.604126:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12500 00000100:00100000:3.0:1734016140.604131:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335396352:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.604139:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.604178:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.604199:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 7, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.604201:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 80 00000020:00010000:3.0:1734016140.604204:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016140.604205:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 80 00000100:00100000:3.0:1734016140.604245:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335396352:12345-192.168.201.116@tcp:106: Request processed in 113us (532us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.604250:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12500 00000100:00100000:3.0:1734016140.617270:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335396480 00000100:00100000:3.0:1734016140.617278:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335396480 00000100:00100000:3.0:1734016140.617418:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12501 00000100:00100000:3.0:1734016140.617427:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335396480:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016140.617444:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6d41 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3387392->3391487) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711701f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.617520:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016140.617525:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335396480:12345-192.168.201.116@tcp:104: Request processed in 100us (358us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.617531:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12501 00000100:00100000:3.0:1734016140.631713:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333933056:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016140.637709:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335396608 00000100:00100000:3.0:1734016140.637717:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335396608 00000100:00100000:3.0:1734016140.638200:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12502 00000100:00100000:3.0:1734016140.638207:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335396608:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016140.638219:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3776512->3784703] (req 3776512->3784703) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.638228:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3776512->3784703] (req 3776512->3784703) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.638236:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.638247:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.638255:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.638264:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.638295:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a781cd80) 00000100:00100000:3.0:1734016140.638339:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335396608:12345-192.168.201.116@tcp:105: Request processed in 132us (748us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.638345:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12502 00000100:00100000:3.0:1734016140.646861:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.656728:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.665485:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.694183:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.700624:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333934336:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.709749:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016140.845170:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335397376 00000100:00100000:3.0:1734016140.845178:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335397376 00000100:00100000:3.0:1734016140.845231:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12508 00000100:00100000:3.0:1734016140.845239:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335397376:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.845254:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.845262:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.845286:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.845288:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 184 00000020:00010000:3.0:1734016140.845290:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016140.845291:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 184 00000100:00100000:3.0:1734016140.845337:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335397376:12345-192.168.201.116@tcp:106: Request processed in 97us (1625us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.845342:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12508 00010000:00010000:3.0:1734016140.855560:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.855576:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.855588:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.855602:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.855608:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.855620:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5535168, setting rss=5535168; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.855642:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.855648:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.855655:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.855661:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.855666:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.855675:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.855679:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.855685:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e6db8 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.859594:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333934720:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.868501:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335397504 00000100:00100000:3.0:1734016140.868509:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335397504 00000100:00100000:3.0:1734016140.868918:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12509 00000100:00100000:3.0:1734016140.868925:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335397504:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.868938:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.868947:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.868973:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.868975:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 184 00000020:00010000:3.0:1734016140.868977:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016140.868978:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 184 00000100:00100000:3.0:1734016140.869446:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335397504:12345-192.168.201.116@tcp:106: Request processed in 520us (1160us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.869453:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12509 00010000:00010000:3.0:1734016140.876123:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.876149:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.876162:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.876168:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.876174:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.876186:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5535168, setting rss=5535168; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.876196:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.876202:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.876208:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.876214:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.876219:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.876228:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.876234:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.876240:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dbf lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.876285:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333934848:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.885017:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335397632 00000100:00100000:3.0:1734016140.885025:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335397632 00000100:00100000:3.0:1734016140.885059:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12510 00000100:00100000:3.0:1734016140.885066:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335397632:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.885081:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.885091:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.885115:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.885117:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 184 00000020:00010000:3.0:1734016140.885120:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016140.885121:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 184 00000100:00100000:3.0:1734016140.885179:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335397632:12345-192.168.201.116@tcp:106: Request processed in 112us (270us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.885185:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12510 00010000:00010000:3.0:1734016140.893773:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.893790:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.893804:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.893811:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.893816:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.893829:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5535168, setting rss=5535168; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.893850:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.893855:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.893862:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.893868:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.893873:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.893883:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.893888:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.893894:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e6dc6 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.895947:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333934976:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.902759:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335397760 00000100:00100000:3.0:1734016140.902768:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335397760 00000100:00100000:3.0:1734016140.902803:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12511 00000100:00100000:3.0:1734016140.902810:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335397760:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016140.902825:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781cd80/0xa50fd972cb8e6d8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3776512->3784703) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711706c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.902891:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016140.902897:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335397760:12345-192.168.201.116@tcp:104: Request processed in 87us (654us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.902903:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12511 00000100:00100000:3.0:1734016140.911977:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333935104:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016140.912001:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2637824->2641919] (req 2637824->2641919) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.912009:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2637824->2641919] (req 2637824->2641919) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.912016:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2637824->2641919] (req 2637824->2641919) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.916021:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335397888 00000100:00100000:3.0:1734016140.916030:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335397888 00000100:00100000:3.0:1734016140.916064:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12512 00000100:00100000:3.0:1734016140.916070:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:9876:x1818247335397888:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016140.916083:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2637824->2641919] (req 2637824->2641919) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.916092:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2637824->2641919] (req 2637824->2641919) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.916099:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2637824->2641919) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.916108:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2637824->2641919) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.916115:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2637824->2641919) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.916124:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2637824->2641919) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.916176:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366f40) 00000100:00100000:3.0:1734016140.916219:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:9876:x1818247335397888:12345-192.168.201.116@tcp:105: Request processed in 150us (278us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.916225:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12512 00000100:00100000:3.0:1734016140.925151:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333935616:192.168.201.116@tcp:4:fsx.0 00000008:00010000:3.0:1734016140.929863:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2637824->2641919) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.929896:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2637824->2641919) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.929912:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333935616:192.168.201.116@tcp:4:fsx.0 00000100:00100000:3.0:1734016140.935207:0:1983:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0000-osc-ffff88012a571000: committing for last_committed 4294968061 gen 2 00000100:00100000:3.0:1734016140.935252:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333935744:192.168.201.116@tcp:16:fsx.0 00000008:00010000:3.0:1734016140.936480:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.936502:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (69632 73727) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016140.936527:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981500 - object 0x280000400:8 offset 69632<>73728 00000100:00100000:3.0:1734016140.936591:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016140.936601:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333936000:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016140.939627:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.939738:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.939758:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333936000:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016140.939778:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee76c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.939802:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (77824 81919) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016140.939825:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981500 - object 0x280000400:8 offset 77824<>81920 00000100:00100000:3.0:1734016140.939848:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016140.939856:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333936256:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016140.943432:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee76c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.943460:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee76c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.943517:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333936256:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016140.948566:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333936384:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016140.952818:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333936512:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016140.953540:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333936640:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.958042:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016140.958055:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335398016 00000100:00100000:3.0:1734016140.958062:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335398016 00000100:00100000:3.0:1734016140.958097:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12513 00000100:00100000:3.0:1734016140.958103:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3ec50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335398016:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.958116:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.958124:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.958154:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 10, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.958156:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 80 00000020:00010000:3.0:1734016140.958159:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016140.958160:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 80 00000100:00100000:3.0:1734016140.958196:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3ec50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335398016:12345-192.168.201.116@tcp:106: Request processed in 93us (458us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.958202:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12513 00010000:00010000:3.0:1734016140.961310:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.961322:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.961333:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.961338:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.961344:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016140.961355:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016140.961361:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.961367:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.961374:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.961380:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016140.961385:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.961394:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016140.961399:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.961404:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6ddb lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016140.961441:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333936768:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.973272:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12517 00000100:00100000:3.0:1734016140.973278:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335398656:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016140.973284:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.973290:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016140.973302:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 10, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016140.973303:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 80 00000020:00010000:3.0:1734016140.973306:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016140.973307:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016140 atime: 1734016140 ctime: 1734016140 blocks: 80 00000100:00100000:3.0:1734016140.973323:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335398656:12345-192.168.201.116@tcp:106: Request processed in 46us (211us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.973328:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12517 00000100:00100000:3.0:1734016140.979883:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335398784 00000100:00100000:3.0:1734016140.979892:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335398784 00000100:00100000:3.0:1734016140.979926:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12518 00000100:00100000:3.0:1734016140.979932:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335398784:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016140.979945:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e6dcd lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2637824->2641919) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271170ab expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.979996:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016140.980001:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335398784:12345-192.168.201.116@tcp:104: Request processed in 70us (553us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.980006:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12518 00000100:00100000:3.0:1734016140.984967:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335398912 00000100:00100000:3.0:1734016140.984974:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335398912 00000100:00100000:3.0:1734016140.985011:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12519 00000100:00100000:3.0:1734016140.985018:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335398912:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016140.985027:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781dd40/0xa50fd972cb8e6dfe lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [819200->831487] (req 819200->831487) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271170dc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.985033:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781dd40/0xa50fd972cb8e6dfe lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [819200->831487] (req 819200->831487) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271170dc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.985040:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781dd40/0xa50fd972cb8e6dfe lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 819200->831487) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271170dc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.985046:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781dd40/0xa50fd972cb8e6dfe lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 819200->831487) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271170dc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.985052:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781dd40/0xa50fd972cb8e6dfe lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 819200->831487) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271170dc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.985060:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781dd40/0xa50fd972cb8e6dfe lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 819200->831487) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271170dc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016140.985084:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a781dd40) 00000100:00100000:3.0:1734016140.985124:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335398912:12345-192.168.201.116@tcp:105: Request processed in 106us (404us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016140.985130:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12519 00000100:00100000:3.0:1734016140.992560:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016140.992577:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333937792:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016140.995683:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.005054:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.014413:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.023451:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.030947:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.041920:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.043988:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016141.048918:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.048930:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.048938:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.048944:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.048950:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.048958:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5535168, setting rss=5535168; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.048966:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.048972:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.048977:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.048982:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.048985:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.048993:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.048996:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.049001:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e6e21 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.049030:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333938560:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.054758:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016141.054768:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335399936 00000100:00100000:3.0:1734016141.054771:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335399936 00000100:00100000:3.0:1734016141.055700:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12527 00000100:00100000:3.0:1734016141.055705:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335399936:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.055714:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.055721:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.055735:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 9, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.055737:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016140 ctime: 1734016141 blocks: 248 00000020:00010000:3.0:1734016141.055739:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016141.055740:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016140 atime: 1734016140 ctime: 1734016141 blocks: 248 00000100:00100000:3.0:1734016141.055760:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335399936:12345-192.168.201.116@tcp:106: Request processed in 56us (2757us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.055767:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12527 00010000:00010000:3.0:1734016141.060116:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060135:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060159:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060166:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060172:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.060188:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.060195:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060201:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060208:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060215:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.060220:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060230:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.060236:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060242:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e6e36 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.060286:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333938944:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.060305:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060312:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060319:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060325:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060330:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.060337:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5535168, setting rss=5535168; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.060346:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060352:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060357:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060362:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.060364:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060370:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.060373:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.060378:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e6e2f lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.060391:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333938816:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016141.065642:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.066061:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.066089:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333939200:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016141.067386:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.067419:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3624960 3629055) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016141.067449:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982680 - object 0x240000400:6 offset 3624960<>3629056 00000008:00010000:3.0:1734016141.072322:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.072351:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.072372:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333939456:192.168.201.116@tcp:3:fsx.0 00000080:00200008:3.0:1734016141.072922:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016141.072928:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:3.0:1734016141.072943:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016141.072955:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016141.072964:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016141.072972:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:3.0:1734016141.072975:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e6b57 00000080:00010000:3.0:1734016141.072980:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6b57 from it ffff8801369fbcd8 00010000:00010000:3.0:1734016141.072982:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016141.072988:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e6b57 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e2e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016141.073002:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016141.073010:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016141.073024:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.073027:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41b0, osc lock ffff8800a8a27560, flags 0x400001002 00000008:00010000:3.0:1734016141.073032:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.073033:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41f0, osc lock ffff8800a8a27690, flags 0x400001002 00010000:00010000:3.0:1734016141.073041:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073047:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073055:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073061:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073071:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073081:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073087:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073092:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016141.073105:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9729472 i_blocks 328 00000020:00010000:3.0:1734016141.073107:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef478 (ffff88012dc12008/0) at cl_lock_release():233 00000020:00010000:3.0:1734016141.073109:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef478 (ffff88012dc12008/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.073111:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41b0 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.073113:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41f0 (ffff88012bf7f3f8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.073114:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef478 (ffff88012dc12008/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016141.073116:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41b0 (ffff88012dc12008/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016141.073118:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41f0 (ffff88012dc12008/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016141.073124:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016141.073133:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016141.073147:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016141.073150:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016141.073154:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.073155:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41b0, osc lock ffff8800a8a27690, flags 0x400001002 00000008:00010000:3.0:1734016141.073157:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.073158:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41f0, osc lock ffff8800a8a27560, flags 0x400001002 00010000:00010000:3.0:1734016141.073163:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073170:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073176:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073182:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073190:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073195:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073202:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073207:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016141.073216:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9729472 i_blocks 328 00000020:00010000:3.0:1734016141.073218:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef478 (ffff88012dc12008/0) at cl_lock_release():233 00000020:00010000:3.0:1734016141.073219:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef478 (ffff88012dc12008/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.073221:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41b0 (ffff88012bf7f3f8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.073223:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41f0 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.073225:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef478 (ffff88012dc12008/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016141.073226:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41b0 (ffff88012dc12008/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016141.073229:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41f0 (ffff88012dc12008/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016141.073233:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016141.073237:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9729472 00000080:00a00000:3.0:1734016141.073888:0:10406:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800b61fabd0 start=0x7f17cf5cb000 end=0x7f17cf5cc000 vm_flags=0xfb 00000080:00200000:3.0:1734016141.073896:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016141.073900:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016141.073904:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.073906:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41b0, osc lock ffff8800a8a27560, flags 0x400001002 00000008:00010000:3.0:1734016141.073908:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.073910:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41f0, osc lock ffff8800a8a27690, flags 0x400001002 00010000:00010000:3.0:1734016141.073916:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073923:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073929:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073936:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073946:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073951:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073956:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.073962:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016141.073969:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9729472 i_blocks 328 00000020:00010000:3.0:1734016141.073971:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef478 (ffff88012dc12008/0) at cl_lock_release():233 00000020:00010000:3.0:1734016141.073973:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef478 (ffff88012dc12008/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.073975:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41b0 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.073977:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41f0 (ffff88012bf7f3f8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.073979:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef478 (ffff88012dc12008/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016141.073980:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41b0 (ffff88012dc12008/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016141.073982:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41f0 (ffff88012dc12008/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016141.073986:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016141.074027:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 1769 00000080:00200000:3.0:1734016141.074033:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016141.074039:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016141.074047:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1769, 1769] 00000008:00010000:3.0:1734016141.074053:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016141.074055:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58ba230, osc lock ffff8800a8a27690, flags 0x0 00010000:00010000:3.0:1734016141.074061:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.074067:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3051520 3055615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.074076:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.074081:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.074086:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.074147:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:3.0:1734016141.074164:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8801369d4180 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.074184:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3051520 3055615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016141.074212:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982680 - object 0x280000400:8 offset 3051520<>3055616 00000100:00100000:3.0:1734016141.074233:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333939712:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016141.076840:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8801369d4180 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.076870:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8801369d4180 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.076887:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333939712:192.168.201.116@tcp:3:fsx.0 00000080:00200000:3.0:1734016141.092693:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016141.092705:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016141.092723:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.092726:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41b0, osc lock ffff8800a8a27560, flags 0x400001002 00000008:00010000:3.0:1734016141.092731:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.092733:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41f0, osc lock ffff8800a8a270a0, flags 0x400001002 00010000:00010000:3.0:1734016141.092753:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34ba80/0xa50fd972cb8e6e3d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.092760:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34ba80/0xa50fd972cb8e6e3d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.092802:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016141.092812:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333940096:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.093291:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3498c0/0xa50fd972cb8e6e44 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.093299:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3498c0/0xa50fd972cb8e6e44 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.098368:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.101690:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016141.106716:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9729472 i_blocks 360 00000020:00010000:3.0:1734016141.106721:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_release():233 00000020:00010000:3.0:1734016141.106724:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.106728:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41b0 (ffff88012bf7f488/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.106731:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41f0 (ffff88012bf7f368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.106733:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016141.106735:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41b0 (ffff88012dc12128/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016141.106738:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41f0 (ffff88012dc12128/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016141.106748:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016141.106767:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016141.106771:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016141.106778:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016141.106792:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.106794:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41b0, osc lock ffff8800a8a270a0, flags 0x400001002 00000008:00010000:3.0:1734016141.106798:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.106800:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41f0, osc lock ffff8800a8a27560, flags 0x400001002 00010000:00010000:3.0:1734016141.106818:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34aac0/0xa50fd972cb8e6e4b lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.106825:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34aac0/0xa50fd972cb8e6e4b lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.106862:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a640/0xa50fd972cb8e6e52 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.106867:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a640/0xa50fd972cb8e6e52 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.110738:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.111003:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.111496:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12531 00000100:00100000:3.0:1734016141.111501:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335400576:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.111507:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.111517:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.111550:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 12, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.111551:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 248 00000020:00010000:3.0:1734016141.111554:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016141.111555:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5535168 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 248 00000100:00100000:3.0:1734016141.111715:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335400576:12345-192.168.201.116@tcp:106: Request processed in 212us (712us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.111720:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12531 00000080:00200000:3.0:1734016141.114872:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9729472 i_blocks 360 00000020:00010000:3.0:1734016141.114879:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_release():233 00000020:00010000:3.0:1734016141.114882:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.114887:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41b0 (ffff88012bf7f368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.114890:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41f0 (ffff88012bf7f488/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016141.114892:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016141.114894:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41b0 (ffff88012dc12128/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016141.114898:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41f0 (ffff88012dc12128/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016141.114908:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016141.114919:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9729472 00000080:00200008:3.0:1734016141.114934:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016141.114939:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016141.114953:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016141.114965:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6b5e lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e3c expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016141.114978:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6b5e lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e3c expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016141.114987:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:3.0:1734016141.114989:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e6b5e 00000080:00010000:3.0:1734016141.114994:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6b5e from it ffff8801369fbcd8 00010000:00010000:3.0:1734016141.114996:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6b5e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27116e3c expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016141.115003:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e6b5e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27116e3c expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016141.115021:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016141.115031:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016141.115048:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.115051:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41b0, osc lock ffff8800a8a27560, flags 0x400001002 00000008:00010000:3.0:1734016141.115056:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016141.115057:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41f0, osc lock ffff8800a8a270a0, flags 0x400001002 00010000:00010000:3.0:1734016141.115075:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.115081:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.115122:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa349440/0xa50fd972cb8e6e60 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.115126:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa349440/0xa50fd972cb8e6e60 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.119178:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.119227:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.119521:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12533 00000100:00100000:3.0:1734016141.119526:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335400832:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.119533:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.119539:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.119558:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 12, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.119561:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 112 00000020:00010000:3.0:1734016141.119564:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.119565:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 112 00000100:00100000:3.0:1734016141.119604:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335400832:12345-192.168.201.116@tcp:106: Request processed in 77us (376us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.119610:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12533 00010000:00010000:3.0:1734016141.123634:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.123645:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.123653:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.123664:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.123670:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.123680:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5535168, setting rss=5535168; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.123696:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.123702:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.123708:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.123713:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa348fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.123716:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.123724:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.123727:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.123733:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e6e59 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.123763:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333940608:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.128943:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016141.128959:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335400960 00000100:00100000:3.0:1734016141.128966:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335400960 00000100:00100000:3.0:1734016141.129006:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12534 00000100:00100000:3.0:1734016141.129012:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2df850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335400960:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.129026:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.129037:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ced lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4009984->4018175) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27116fcb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.129057:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 12, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.129059:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 112 00000020:00010000:3.0:1734016141.129062:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.129063:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 112 00000100:00100000:3.0:1734016141.129103:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2df850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335400960:12345-192.168.201.116@tcp:106: Request processed in 90us (1473us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.129109:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12534 00000100:00100000:3.0:1734016141.143461:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335401216 00000100:00100000:3.0:1734016141.143527:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335401216 00000100:00100000:3.0:1734016141.143568:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12535 00000100:00100000:3.0:1734016141.143575:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335401216:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016141.143592:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e6e1a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3940352->3948543) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271170f8 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.143655:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016141.143660:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335401216:12345-192.168.201.116@tcp:104: Request processed in 87us (294us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.143666:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12535 00000100:00100000:3.0:1734016141.149548:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335401344 00000100:00100000:3.0:1734016141.149556:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335401344 00000100:00100000:3.0:1734016141.149597:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12536 00000100:00100000:3.0:1734016141.149602:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:9876:x1818247335401344:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016141.149618:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e6e75 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1028096->1036287] (req 1028096->1036287) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117153 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.149627:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e6e75 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1028096->1036287] (req 1028096->1036287) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117153 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.149635:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e6e75 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1028096->1036287) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117153 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.149645:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e6e75 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1028096->1036287) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117153 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.149652:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e6e75 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1028096->1036287) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117153 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.149661:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e6e75 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1028096->1036287) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117153 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.149690:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c2400) 00000100:00100000:3.0:1734016141.149740:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:9876:x1818247335401344:12345-192.168.201.116@tcp:105: Request processed in 137us (339us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.149747:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12536 00000100:00100000:3.0:1734016141.226766:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.238387:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.238438:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.238460:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335401600 00000100:00100000:3.0:1734016141.238493:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335401600 00000100:00100000:3.0:1734016141.238521:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335401728 00000100:00100000:3.0:1734016141.238523:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335401728 00000100:00100000:3.0:1734016141.238534:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12538 00000100:00100000:3.0:1734016141.238540:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335401600:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016141.238553:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f3c0/0xa50fd972cb8e6e7c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc2711715a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.238605:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016141.238610:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335401600:12345-192.168.201.116@tcp:104: Request processed in 71us (231us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.238617:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12538 00000100:00100000:3.0:1734016141.238628:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12539 00000100:00100000:3.0:1734016141.238631:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335401728:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016141.238638:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e6e75 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1028096->1036287) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117153 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.239239:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016141.239245:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335401728:12345-192.168.201.116@tcp:104: Request processed in 612us (806us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.239250:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12539 00000100:00100000:3.0:1734016141.248582:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.250994:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.257825:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333942528:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.257849:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6e8a lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711716f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.257858:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6e8a lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711716f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.257864:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6e8a lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711716f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.259620:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.269046:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333942912:192.168.201.116@tcp:10:fsx.0 00000100:00100000:3.0:1734016141.296406:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.296608:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.306220:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.306317:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016141.312374:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.312390:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.312403:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.312409:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.312415:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.312430:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2754863, setting rss=2754863; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.312439:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.312445:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.312450:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.312455:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.312460:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.312493:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.312500:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.312505:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6ea6 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.313392:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333943424:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.319359:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333943680:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016141.322760:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016141.322774:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335402752 00000100:00100000:3.0:1734016141.322780:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335402752 00000100:00100000:3.0:1734016141.322808:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12547 00000100:00100000:3.0:1734016141.322814:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012ab3c050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335402752:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.322829:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6e83 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117168 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.322837:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6e83 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117168 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.322859:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.322861:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2754863 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 120 00000020:00010000:3.0:1734016141.322864:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016141.322865:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2754863 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 120 00000100:00100000:3.0:1734016141.322900:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012ab3c050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335402752:12345-192.168.201.116@tcp:106: Request processed in 87us (263us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.322906:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12547 00010000:00010000:3.0:1734016141.327060:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327076:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327088:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327094:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327099:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.327111:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.327118:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327123:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327128:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327147:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.327151:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327162:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.327167:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327172:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e6ebb lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.327212:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333943936:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.327230:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327237:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327243:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327248:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327254:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.327260:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2754863, setting rss=2754863; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.327272:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327277:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327282:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327286:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.327289:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327294:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.327296:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.327301:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e6eb4 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.327312:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333943808:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.333272:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333292:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333304:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333310:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333315:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.333329:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2754863, setting rss=2754863; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.333337:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333342:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333348:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333353:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.333359:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333371:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.333376:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333381:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e6ec2 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.333426:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333944064:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.333441:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333447:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333453:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333458:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333463:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.333633:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.333647:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333654:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333659:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333663:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.333666:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333671:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.333673:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.333679:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6ec9 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.333692:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333944192:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.339925:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.339942:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.339955:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.339961:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.339967:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.339979:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2754863, setting rss=2754863; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.339986:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.339991:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.339998:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340006:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.340012:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340023:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.340029:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340033:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6ed0 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.340061:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333944320:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.340077:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340084:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340090:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340096:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340101:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.340108:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.340125:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340139:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340145:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340154:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.340158:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340163:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.340166:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.340171:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e6ed7 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.340186:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333944448:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.343652:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335403520 00000100:00100000:3.0:1734016141.343660:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335403520 00000100:00100000:3.0:1734016141.343693:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12552 00000100:00100000:3.0:1734016141.343700:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335403520:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016141.343714:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e6e83 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117168 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.343767:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016141.343772:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335403520:12345-192.168.201.116@tcp:104: Request processed in 74us (393us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.343777:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12552 00000100:00100000:3.0:1734016141.350997:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335403648 00000100:00100000:3.0:1734016141.351006:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335403648 00000100:00100000:3.0:1734016141.351041:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12553 00000100:00100000:3.0:1734016141.351048:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335403648:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016141.351061:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [6094848->6107135] (req 6094848->6107135) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.351071:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [6094848->6107135] (req 6094848->6107135) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.351079:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.351089:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.351096:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.351105:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.351125:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366ac0) 00000100:00100000:3.0:1734016141.351174:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335403648:12345-192.168.201.116@tcp:105: Request processed in 126us (529us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.351181:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12553 00000100:00100000:3.0:1734016141.364980:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.369276:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333945216:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.373832:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.381232:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.388760:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.393705:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335404288 00000100:00100000:3.0:1734016141.393711:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335404288 00000100:00100000:3.0:1734016141.393750:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12558 00000100:00100000:3.0:1734016141.393757:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6466:x1818247335404288:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016141.393765:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [65536->77823] (req 65536->77823) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.393772:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [65536->77823] (req 65536->77823) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.393778:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.393785:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.393792:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.393799:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.393814:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa367cc0) 00000100:00100000:3.0:1734016141.393857:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6466:x1818247335404288:12345-192.168.201.116@tcp:105: Request processed in 99us (575us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.393863:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12558 00010000:00010000:3.0:1734016141.403921:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.403936:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.403948:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.403958:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.403964:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.403976:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 74129, setting rss=74129; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.403982:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.403987:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.403993:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.403998:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.404001:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.404009:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.404013:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.404018:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e6f0f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.404059:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333945984:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.407302:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.407317:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.407329:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.407335:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.407340:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.407353:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2754863, setting rss=2754863; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.407373:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.407379:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.407386:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.407392:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.407397:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.407406:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.407411:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.407416:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f08 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.407439:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333945856:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.411610:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016141.411625:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335404672 00000100:00100000:3.0:1734016141.411634:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335404672 00000100:00100000:3.0:1734016141.411677:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12561 00000100:00100000:3.0:1734016141.411684:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f858050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335404672:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.411697:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.411711:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.411732:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.411734:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 74129 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 24 00000020:00010000:3.0:1734016141.411737:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.411738:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 74129 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 24 00000100:00100000:3.0:1734016141.411774:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f858050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335404672:12345-192.168.201.116@tcp:106: Request processed in 91us (876us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.411780:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12561 00010000:00010000:3.0:1734016141.415915:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.415930:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.415942:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.415947:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.415953:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.415968:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 74129, setting rss=74129; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.415976:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.415982:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.415988:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.415993:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.415998:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.416006:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.416011:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.416016:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f1d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.416699:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333946240:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.427055:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.427238:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.427258:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335404928 00000100:00100000:3.0:1734016141.427264:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335404928 00000100:00100000:3.0:1734016141.427740:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12562 00000100:00100000:3.0:1734016141.427747:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335404928:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.427760:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.427770:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.427791:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.427793:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 74129 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 24 00000020:00010000:3.0:1734016141.427796:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.427797:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 74129 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 24 00000100:00100000:3.0:1734016141.427834:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335404928:12345-192.168.201.116@tcp:106: Request processed in 87us (786us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.427840:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12562 00000100:00100000:3.0:1734016141.427852:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335405056 00000100:00100000:3.0:1734016141.427854:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335405056 00010000:00010000:3.0:1734016141.431097:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431122:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431144:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431152:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431159:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.431175:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2754863, setting rss=2754863; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.431185:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431191:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431198:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431204:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.431209:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431221:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.431227:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431233:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e6f24 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.431923:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333946496:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.431943:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431954:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431960:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431966:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.431971:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.431978:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 74129, setting rss=74129; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.432020:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.432026:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.432032:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.432036:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.432039:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.432049:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.432052:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.432057:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e6f2b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.432079:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333946624:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.436890:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335405184 00000100:00100000:3.0:1734016141.436898:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335405184 00000100:00100000:3.0:1734016141.436944:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12564 00000100:00100000:3.0:1734016141.436950:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335405184:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.436964:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.436975:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.437000:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.437002:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2754863 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 152 00000020:00010000:3.0:1734016141.437005:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016141.437006:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2754863 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 152 00000100:00100000:3.0:1734016141.437058:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335405184:12345-192.168.201.116@tcp:106: Request processed in 107us (397us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.437065:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12564 00000100:00100000:3.0:1734016141.439501:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335405312 00000100:00100000:3.0:1734016141.439509:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335405312 00000100:00100000:3.0:1734016141.439537:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12565 00000100:00100000:3.0:1734016141.439542:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335405312:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.439554:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.439563:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.439585:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.439586:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 74129 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 24 00000020:00010000:3.0:1734016141.439589:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.439590:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 74129 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 24 00000100:00100000:3.0:1734016141.439626:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335405312:12345-192.168.201.116@tcp:106: Request processed in 84us (633us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.439631:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12565 00010000:00010000:3.0:1734016141.443898:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.443915:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.443927:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.443933:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.443940:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.443955:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 74129, setting rss=74129; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.443961:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.443966:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.443973:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.443979:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.443984:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.443993:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.443997:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.444003:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e6f39 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.444042:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333946880:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.444060:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.444067:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.444073:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.444079:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.444085:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.444091:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2754863, setting rss=2754863; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.444106:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.444113:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.444118:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.444123:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.444126:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.444140:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.444143:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.444148:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e6f32 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.444161:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333946752:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.450343:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333947008:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016141.455024:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335405440 00000100:00100000:3.0:1734016141.455032:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335405440 00000100:00100000:3.0:1734016141.455060:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335405568 00000100:00100000:3.0:1734016141.455063:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335405568 00000100:00100000:3.0:1734016141.455074:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12566 00000100:00100000:3.0:1734016141.455080:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335405440:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.455093:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.455104:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.455125:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.455127:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 74129 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 24 00000020:00010000:3.0:1734016141.455139:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.455141:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 74129 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 24 00000100:00100000:3.0:1734016141.455185:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335405440:12345-192.168.201.116@tcp:106: Request processed in 104us (514us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.455192:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12566 00000100:00100000:3.0:1734016141.455204:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12567 00000100:00100000:3.0:1734016141.455208:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335405568:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.455214:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.455219:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.455229:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.455230:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2754863 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 152 00000020:00010000:3.0:1734016141.455233:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016141.455234:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2754863 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 152 00000100:00100000:3.0:1734016141.455248:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335405568:12345-192.168.201.116@tcp:106: Request processed in 39us (344us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.455252:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12567 00010000:00010000:3.0:1734016141.458541:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.458556:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.458568:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.458579:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.458585:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.458599:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2754863, setting rss=2754863; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.458605:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.458610:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.458616:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.458622:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.458627:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.458637:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.458641:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.458647:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e6f40 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.459230:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333947136:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.465828:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.465903:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.475067:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335405952 00000100:00100000:3.0:1734016141.475075:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335405952 00000100:00100000:3.0:1734016141.475113:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12569 00000100:00100000:3.0:1734016141.475119:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335405952:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.475139:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.475149:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.475170:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.475172:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2754863 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 152 00000020:00010000:3.0:1734016141.475175:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016141.475176:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2754863 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 152 00000100:00100000:3.0:1734016141.475214:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335405952:12345-192.168.201.116@tcp:106: Request processed in 95us (348us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.475220:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12569 00000100:00100000:3.0:1734016141.484940:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335406208 00000100:00100000:3.0:1734016141.484948:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335406208 00000100:00100000:3.0:1734016141.485301:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12571 00000100:00100000:3.0:1734016141.485309:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335406208:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.485321:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.485331:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6ede lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6094848->6107135) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171ca expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.485353:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.485355:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2754863 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 152 00000020:00010000:3.0:1734016141.485358:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016141.485359:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2754863 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 152 00000100:00100000:3.0:1734016141.485397:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335406208:12345-192.168.201.116@tcp:106: Request processed in 88us (695us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.485402:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12571 00000100:00100000:3.0:1734016141.486373:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335406336 00000100:00100000:3.0:1734016141.486378:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335406336 00000100:00100000:3.0:1734016141.486438:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12572 00000100:00100000:3.0:1734016141.486443:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335406336:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.486452:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.486460:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.486500:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.486502:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 74129 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 24 00000020:00010000:3.0:1734016141.486504:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.486505:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 74129 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 24 00000100:00100000:3.0:1734016141.486533:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335406336:12345-192.168.201.116@tcp:106: Request processed in 88us (740us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.486538:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12572 00000100:00100000:3.0:1734016141.508386:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.515782:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.520458:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016141.520490:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335406720 00000100:00100000:3.0:1734016141.520495:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335406720 00000100:00100000:3.0:1734016141.520525:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12575 00000100:00100000:3.0:1734016141.520531:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f859050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335406720:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.520544:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.520553:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.520574:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.520576:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2136811 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 32 00000020:00010000:3.0:1734016141.520579:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.520580:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2136811 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 32 00000100:00100000:3.0:1734016141.520615:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f859050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335406720:12345-192.168.201.116@tcp:106: Request processed in 84us (371us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.520621:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12575 00000100:00100000:3.0:1734016141.591803:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.604723:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12578 00000100:00100000:3.0:1734016141.604728:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335407104:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016141.604735:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781d200/0xa50fd972cb8e6f78 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3710976->3719167) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711726b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.606940:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016141.606950:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335407104:12345-192.168.201.116@tcp:104: Request processed in 2218us (2424us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.606957:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12578 00000100:00100000:3.0:1734016141.615985:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.629117:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335407360 00000100:00100000:3.0:1734016141.629126:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335407360 00000100:00100000:3.0:1734016141.629702:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12580 00000100:00100000:3.0:1734016141.629710:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335407360:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016141.629724:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e6f01 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 65536->77823) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271171f4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.629821:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016141.629827:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335407360:12345-192.168.201.116@tcp:104: Request processed in 118us (1407us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.629834:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12580 00000100:00100000:3.0:1734016141.635208:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016141.635219:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335407488 00000100:00100000:3.0:1734016141.635224:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335407488 00000100:00100000:3.0:1734016141.635252:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12581 00000100:00100000:3.0:1734016141.635257:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2ddc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:9876:x1818247335407488:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016141.635266:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.635273:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.635284:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.635291:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.635297:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.635304:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.635316:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c0fc0) 00000100:00100000:3.0:1734016141.635349:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2ddc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:9876:x1818247335407488:12345-192.168.201.116@tcp:105: Request processed in 92us (290us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.635354:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12581 00000100:00100000:3.0:1734016141.649288:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.655308:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.741121:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335407872 00000100:00100000:3.0:1734016141.741138:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335407872 00000100:00100000:3.0:1734016141.741178:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12584 00000100:00100000:3.0:1734016141.741185:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335407872:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.741199:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.741212:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.741234:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.741236:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000020:00010000:3.0:1734016141.741239:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.741240:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000100:00100000:3.0:1734016141.741281:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335407872:12345-192.168.201.116@tcp:106: Request processed in 96us (443us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.741287:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12584 00000100:00100000:3.0:1734016141.752429:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.766617:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.775731:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000008:00010000:3.0:1734016141.798988:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7780 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e6fbe lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc271172c6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.799011:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1503232 1507327) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e6fbe lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc271172c6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016141.799037:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981880 - object 0x240000400:6 offset 1503232<>1507328 00000008:00010000:3.0:1734016141.806963:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7780 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e6fbe lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc271172c6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.807312:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7780 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e6fbe lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc271172c6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.807341:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333952640:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016141.812756:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016141.812770:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335408384 00000100:00100000:3.0:1734016141.812776:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335408384 00000100:00100000:3.0:1734016141.812812:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12588 00000100:00100000:3.0:1734016141.812817:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dfc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335408384:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.812833:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.812842:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.812861:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 193, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.812863:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000020:00010000:3.0:1734016141.812866:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.812867:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000100:00100000:3.0:1734016141.812899:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dfc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335408384:12345-192.168.201.116@tcp:106: Request processed in 82us (450us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.812905:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12588 00010000:00010000:3.0:1734016141.815715:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.815731:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.815747:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.815753:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.815760:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.815773:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.815792:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.815798:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.815805:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.815811:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.815815:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.815825:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.815830:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.815840:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e6fd3 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.815878:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333952768:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.823549:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335408512 00000100:00100000:3.0:1734016141.823558:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335408512 00000100:00100000:3.0:1734016141.823605:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12589 00000100:00100000:3.0:1734016141.823611:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335408512:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.823625:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.823634:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.823654:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 193, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.823656:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000020:00010000:3.0:1734016141.823659:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.823660:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000100:00100000:3.0:1734016141.823704:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335408512:12345-192.168.201.116@tcp:106: Request processed in 93us (2100us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.823710:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12589 00010000:00010000:3.0:1734016141.827118:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.827166:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.827179:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.827188:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.827194:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.827207:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.827225:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.827232:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.827238:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.827244:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.827249:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.827259:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.827264:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.827274:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e6fda lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.827307:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333952896:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.831773:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335408640 00000100:00100000:3.0:1734016141.831781:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335408640 00000100:00100000:3.0:1734016141.831816:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12590 00000100:00100000:3.0:1734016141.831822:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:8c598caf-7141-4c19-8566-0b7731516db9+5:6478:x1818247335408640:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016141.831836:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e6fa9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc271172b1 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016141.832133:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016141.832140:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:8c598caf-7141-4c19-8566-0b7731516db9+5:6478:x1818247335408640:12345-192.168.201.116@tcp:104: Request processed in 317us (876us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.832145:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12590 00000100:00100000:3.0:1734016141.844559:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.844612:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.845208:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12592 00000100:00100000:3.0:1734016141.845213:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7982680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335408896:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016141.845220:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e6fa2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 430080->434175) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271172a3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.845531:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016141.845537:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7982680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335408896:12345-192.168.201.116@tcp:104: Request processed in 323us (924us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.845543:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12592 00000100:00100000:3.0:1734016141.854276:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.887310:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.887400:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.899851:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.899900:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016141.928710:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335409664 00000100:00100000:3.0:1734016141.928720:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335409664 00000100:00100000:3.0:1734016141.928766:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12598 00000100:00100000:3.0:1734016141.928772:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335409664:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.928788:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.928798:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.928823:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.928825:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000020:00010000:3.0:1734016141.928828:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016141.928828:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000100:00100000:3.0:1734016141.928874:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335409664:12345-192.168.201.116@tcp:106: Request processed in 103us (1086us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.928879:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12598 00000100:00100000:3.0:1734016141.929458:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335409792 00000100:00100000:3.0:1734016141.929481:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335409792 00000100:00100000:3.0:1734016141.929513:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12599 00000100:00100000:3.0:1734016141.929516:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335409792:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.929524:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.929534:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.929548:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.929549:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 144 00000020:00010000:3.0:1734016141.929552:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016141.929553:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 144 00000100:00100000:3.0:1734016141.929578:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335409792:12345-192.168.201.116@tcp:106: Request processed in 60us (189us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.929582:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12599 00010000:00010000:3.0:1734016141.937983:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.937997:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938020:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938026:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938031:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.938049:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4298598, setting rss=4298598; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.938055:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938061:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938067:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938072:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.938077:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938087:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.938091:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938096:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7012 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.938141:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333954944:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016141.938770:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938783:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938790:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938795:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938800:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016141.938806:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016141.938814:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938820:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938825:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938830:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016141.938833:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938843:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016141.938846:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.938850:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7019 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016141.938863:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333955072:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016141.943880:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335409920 00000100:00100000:3.0:1734016141.943888:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335409920 00000100:00100000:3.0:1734016141.943919:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12600 00000100:00100000:3.0:1734016141.943926:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335409920:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016141.943941:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e6f94 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711728e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.944159:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016141.944166:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335409920:12345-192.168.201.116@tcp:104: Request processed in 240us (382us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.944172:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12600 00000100:00100000:3.0:1734016141.953162:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335410048 00000100:00100000:3.0:1734016141.953168:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335410048 00000100:00100000:3.0:1734016141.953195:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12601 00000100:00100000:3.0:1734016141.953200:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335410048:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016141.953208:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7020 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1073152->1077247] (req 1073152->1077247) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711733d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.953219:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7020 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1073152->1077247] (req 1073152->1077247) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711733d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.953226:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7020 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1073152->1077247) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711733d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.953232:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7020 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1073152->1077247) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711733d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.953238:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7020 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1073152->1077247) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711733d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.953245:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7020 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1073152->1077247) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711733d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.953260:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa367840) 00000100:00100000:3.0:1734016141.953300:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6466:x1818247335410048:12345-192.168.201.116@tcp:105: Request processed in 99us (332us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.953306:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12601 00000100:00100000:3.0:1734016141.960213:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016141.960225:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335410176 00000100:00100000:3.0:1734016141.960229:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335410176 00000100:00100000:3.0:1734016141.960502:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12602 00000100:00100000:3.0:1734016141.960508:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f85ac50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335410176:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016141.960522:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.960531:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016141.960552:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016141.960554:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 144 00000020:00010000:3.0:1734016141.960557:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016141.960558:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 144 00000100:00100000:3.0:1734016141.961243:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f85ac50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335410176:12345-192.168.201.116@tcp:106: Request processed in 733us (1119us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.961251:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12602 00000100:00100000:3.0:1734016141.987487:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335410560 00000100:00100000:3.0:1734016141.987493:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335410560 00000100:00100000:3.0:1734016141.987528:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12605 00000100:00100000:3.0:1734016141.987535:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335410560:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016141.987548:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1982464->1986559] (req 1982464->1986559) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.987559:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1982464->1986559] (req 1982464->1986559) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.987566:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.987576:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.987587:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.987595:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016141.987611:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a781eac0) 00000100:00100000:3.0:1734016141.987656:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335410560:12345-192.168.201.116@tcp:105: Request processed in 120us (1888us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016141.987662:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12605 00000100:00100000:3.0:1734016142.031111:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333957120:192.168.201.116@tcp:4:fsx.0 00000008:00010000:3.0:1734016142.038916:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a7b19240 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.038952:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a7b19240 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.038972:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333957120:192.168.201.116@tcp:4:fsx.0 00010000:00010000:3.0:1734016142.060260:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 6/2,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.060277:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.060688:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:3.0:1734016142.060708:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8801369d4840 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.060762:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2678784 2682879) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016142.060880:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981f80 - object 0x240000400:6 offset 0<>3534848 00000100:00100000:3.0:1734016142.060907:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333957632:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016142.113981:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8801369d4840 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.114249:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8801369d4840 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.114281:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333957632:192.168.201.116@tcp:3:fsx.0 00010000:00010000:3.0:1734016142.127386:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1724416 1728511) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016142.127875:0:10406:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981f80 - object 0x240000400:6 offset 1724416<>1732608 00000100:00100000:3.0:1734016142.127943:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016142.127961:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333957760:192.168.201.116@tcp:4:fsx.0 00000008:00010000:3.0:1734016142.137962:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.139723:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.140732:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333957760:192.168.201.116@tcp:4:fsx.0 00000100:00100000:3.0:1734016142.165831:0:1984:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0000-osc-ffff88012a571000: committing for last_committed 4294968070 gen 2 00010000:00010000:3.0:1734016142.165879:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.165892:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.165905:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.165911:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.165918:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.165932:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4298598, setting rss=4298598; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.165939:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.165944:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.165951:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.165956:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.165962:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.165971:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.165975:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.165981:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e703c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.166020:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333958016:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.166612:0:1984:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff88012a571000: committing for last_committed 4294969534 gen 2 00010000:00010000:3.0:1734016142.166628:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.166635:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.166642:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.166651:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.166656:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.166663:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 1990656, setting rss=1990656; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.166685:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.166690:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.166695:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.166699:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.166701:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.166706:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.166709:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.166713:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7043 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.166729:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333958144:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016142.175857:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.175876:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.175889:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.175895:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.175900:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.175914:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4298598, setting rss=4298598; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.175921:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.175925:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.175932:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.175941:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.175945:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.175953:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.175958:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.175963:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e704a lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.175990:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333958272:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016142.176013:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.176020:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.176027:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.176032:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.176037:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.176044:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 1990656, setting rss=1990656; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.176063:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.176069:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.176074:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.176078:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.176081:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.176086:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.176089:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.176094:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7051 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.176108:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333958400:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016142.187985:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.188137:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.188161:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333958912:192.168.201.116@tcp:3:fsx.0 00010000:00010000:3.0:1734016142.199172:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.199189:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.199202:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.199208:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.199214:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.199226:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4298598, setting rss=4298598; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.199233:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.199238:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.199248:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.199254:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.199259:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.199268:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.199273:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.199279:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7058 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.199308:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333959040:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016142.202226:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.202242:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.202251:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.202256:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.202262:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.202272:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 1990656, setting rss=1990656; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.202816:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.202823:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.202829:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.202834:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.202839:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.202848:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.202852:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.202858:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e705f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.202884:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333959168:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016142.210306:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.210322:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.210334:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.210340:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.210345:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.210357:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 1990656, setting rss=1990656; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.210363:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.210369:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.210375:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.210381:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.210386:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.210395:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.210400:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.210405:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e706d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.210429:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333959424:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016142.211298:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.211308:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.211315:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.211320:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.211325:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.211332:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4298598, setting rss=4298598; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.211382:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.211389:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.211395:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.211400:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.211403:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.211409:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.211412:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.211417:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7066 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.211431:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333959296:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.213362:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016142.213377:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333959552:192.168.201.116@tcp:4:fsx.0 00000008:00010000:3.0:1734016142.218683:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7780 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.218917:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7780 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.218941:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333959552:192.168.201.116@tcp:4:fsx.0 00000100:00100000:3.0:1734016142.219143:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333959680:192.168.201.116@tcp:16:fsx.0 00000100:00100000:3.0:1734016142.225774:0:1984:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff88012a5ff800: committing for last_committed 4294969538 gen 1 00000100:00100000:3.0:1734016142.225817:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333959680:192.168.201.116@tcp:16:fsx.0 00010000:00010000:3.0:1734016142.226974:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.226983:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.226989:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016142.226999:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8492902 i_blocks 200 00000020:00010000:3.0:1734016142.227002:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016142.227003:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.227005:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee77b0 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.227008:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee77f0 (ffff88012dc12a28/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.227010:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.227012:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee77b0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.227015:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee77f0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016142.227020:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016142.227045:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 436 00000080:00200000:3.0:1734016142.227063:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 437 00000080:00200000:3.0:1734016142.227072:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 438 00000080:00200008:3.0:1734016142.227151:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016142.227155:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016142.227164:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016142.227171:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e700b lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117328 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016142.227178:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e700b lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117328 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016142.227184:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:3.0:1734016142.227186:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e700b 00000080:00010000:3.0:1734016142.227189:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e700b from it ffff8801369fbcd8 00010000:00010000:3.0:1734016142.227191:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e700b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117328 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016142.227197:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e700b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117328 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016142.227207:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016142.227211:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016142.227217:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.227219:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4cf0, osc lock ffff8800a8a27300, flags 0x400001002 00000008:00010000:3.0:1734016142.227222:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.227223:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4d30, osc lock ffff8800a8a26f70, flags 0x400001002 00010000:00010000:3.0:1734016142.227229:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227236:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227242:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227247:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227256:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227262:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227267:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227273:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016142.227280:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8492902 i_blocks 200 00000020:00010000:3.0:1734016142.227282:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016142.227284:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.227286:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4cf0 (ffff88012dc12a28/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.227288:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4d30 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.227290:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.227292:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4cf0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.227294:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4d30 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016142.227299:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016142.227305:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016142.227307:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016142.227310:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016142.227313:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.227315:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4cf0, osc lock ffff8800a8a26f70, flags 0x400001002 00000008:00010000:3.0:1734016142.227317:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.227318:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4d30, osc lock ffff8800a8a27300, flags 0x400001002 00010000:00010000:3.0:1734016142.227323:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227329:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227336:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227341:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227350:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227355:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227360:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227365:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016142.227373:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8492902 i_blocks 200 00000020:00010000:3.0:1734016142.227375:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016142.227376:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.227378:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4cf0 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.227381:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4d30 (ffff88012dc12a28/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.227382:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.227384:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4cf0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.227386:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4d30 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016142.227390:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016142.227393:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8492902 00000080:00200008:3.0:1734016142.227401:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016142.227405:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:3.0:1734016142.227413:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016142.227421:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e6fe8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271172fe expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016142.227428:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e6fe8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271172fe expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016142.227434:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:3.0:1734016142.227436:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e6fe8 00000080:00010000:3.0:1734016142.227438:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e6fe8 from it ffff8801369fbcd8 00010000:00010000:3.0:1734016142.227440:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e6fe8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271172fe expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016142.227445:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e6fe8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271172fe expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016142.227455:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016142.227459:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016142.227485:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.227487:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4cf0, osc lock ffff8800a8a27300, flags 0x400001002 00000008:00010000:3.0:1734016142.227490:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.227491:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4d30, osc lock ffff8800a8a26f70, flags 0x400001002 00010000:00010000:3.0:1734016142.227506:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e7074 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227513:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a1c0/0xa50fd972cb8e7074 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.227565:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016142.227577:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333959808:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016142.227643:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34af40/0xa50fd972cb8e707b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.227649:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34af40/0xa50fd972cb8e707b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.227675:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333959936:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.230530:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335411840 00000100:00100000:3.0:1734016142.230534:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335411840 00000100:00100000:3.0:1734016142.230548:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12614 00000100:00100000:3.0:1734016142.230554:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335411712:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016142.230566:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.230574:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016142.230593:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1513, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016142.230595:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 160 00000020:00010000:3.0:1734016142.230597:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016142.230598:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 160 00000100:00100000:3.0:1734016142.230636:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335411712:12345-192.168.201.116@tcp:106: Request processed in 81us (428us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.230642:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12614 00000100:00100000:3.0:1734016142.230654:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12615 00000100:00100000:3.0:1734016142.230657:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335411840:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016142.230663:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.230668:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781eac0/0xa50fd972cb8e7035 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1982464->1986559) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117352 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016142.230678:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1513, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016142.230680:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 1990656 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 40 00000020:00010000:3.0:1734016142.230682:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016142.230683:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 1990656 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 40 00000100:00100000:3.0:1734016142.230698:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335411840:12345-192.168.201.116@tcp:106: Request processed in 40us (282us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.230702:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12615 00000080:00200000:3.0:1734016142.238383:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8492902 i_blocks 200 00000020:00010000:3.0:1734016142.238391:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016142.238394:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.238399:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4cf0 (ffff88012dc12a28/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.238402:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4d30 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.238403:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.238405:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4cf0 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.238408:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4d30 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016142.238419:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016142.238438:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016142.238441:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016142.238459:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016142.238594:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1445, 1447] 00000008:00010000:3.0:1734016142.238609:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016142.238612:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58ba030, osc lock ffff8800a8a26f70, flags 0x0 00010000:00010000:3.0:1734016142.238632:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.238639:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1724416->1736703] (req 1724416->1736703) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.238670:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1724416->1736703] (req 1724416->1736703) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.238689:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333960064:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.238746:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4780 going to sleep for 11 seconds 00000100:00100000:3.0:1734016142.243275:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.254331:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333960064:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016142.254357:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1724416->1736703] (req 1724416->1736703) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.254368:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1724416->1736703] (req 1724416->1736703) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.254375:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1724416->1736703] (req 1724416->1736703) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.255228:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.255237:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.255254:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.255266:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 1990656, setting rss=1990656; kms=1990656, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.255276:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.255286:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016142.255308:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8492902 i_blocks 200 00000100:00100000:3.0:1734016142.269505:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333960320:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016142.272813:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.286057:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.301257:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.314477:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.330826:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200008:3.0:1734016142.365789:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016142.365796:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016142.365814:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016142.365830:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e700b lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117328 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016142.365840:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e700b lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117328 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016142.365848:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:3.0:1734016142.365851:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e700b 00000080:00010000:3.0:1734016142.365857:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e700b from it ffff8801369fbcd8 00010000:00010000:3.0:1734016142.365860:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e700b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117328 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016142.365872:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e700b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117328 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016142.365894:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016142.365905:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016142.365926:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.365929:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d47b0, osc lock ffff8800a8a27300, flags 0x400001002 00000008:00010000:3.0:1734016142.365953:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.365954:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d47f0, osc lock ffff8800a8a271d0, flags 0x400001002 00010000:00010000:3.0:1734016142.365965:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.365972:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.365981:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.365986:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.366006:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34bcc0/0xa50fd972cb8e70ac lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.366013:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34bcc0/0xa50fd972cb8e70ac lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.366064:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016142.366073:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333961600:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.370743:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016142.375939:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8492902 i_blocks 232 00000020:00010000:3.0:1734016142.375947:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbecdb0 (ffff88012dc12a28/0) at cl_lock_release():233 00000020:00010000:3.0:1734016142.375950:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbecdb0 (ffff88012dc12a28/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.375955:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d47b0 (ffff88012bf7f128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.375960:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d47f0 (ffff88012bf7f7e8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.375962:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbecdb0 (ffff88012dc12a28/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.375964:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d47b0 (ffff88012dc12a28/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.375969:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d47f0 (ffff88012dc12a28/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016142.375981:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016142.376001:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016142.376007:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016142.376015:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016142.376031:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.376034:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d47b0, osc lock ffff8800a8a271d0, flags 0x400001002 00000008:00010000:3.0:1734016142.376037:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.376039:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d47f0, osc lock ffff8800a8a27300, flags 0x400001002 00010000:00010000:3.0:1734016142.376053:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.376065:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.376074:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.376081:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e6fe1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271172f0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.376110:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e70b3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.376126:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e70b3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.382586:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016142.389385:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8492902 i_blocks 232 00000020:00010000:3.0:1734016142.389392:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbecdb0 (ffff88012dc12a28/0) at cl_lock_release():233 00000020:00010000:3.0:1734016142.389395:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbecdb0 (ffff88012dc12a28/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.389400:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d47b0 (ffff88012bf7f7e8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.389403:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d47f0 (ffff88012bf7f128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.389405:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbecdb0 (ffff88012dc12a28/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.389407:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d47b0 (ffff88012dc12a28/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.389410:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d47f0 (ffff88012dc12a28/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016142.389418:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016142.389429:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8492902 00000080:00200008:3.0:1734016142.389447:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016142.389453:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:3.0:1734016142.390324:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016142.390340:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e6fe8 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271172fe expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000100:00100000:3.0:1734016142.395680:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.410973:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.422992:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.436806:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.461406:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.478838:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016142.503980:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.503998:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504009:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504015:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504021:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.504035:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2342912, setting rss=2342912; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.504042:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504048:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504055:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504060:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.504065:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504074:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.504078:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504083:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e70e4 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.504124:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333963136:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016142.504138:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504145:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504152:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504156:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504162:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.504169:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4298598, setting rss=4298598; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.504178:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504183:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504188:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504193:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.504196:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504202:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.504204:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.504210:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e70dd lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.504220:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333963008:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016142.513047:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513063:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513074:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513081:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513087:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.513103:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2342912, setting rss=2342912; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.513111:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513127:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513134:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513140:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.513145:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513154:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.513159:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513165:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e70f2 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.513248:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333963392:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016142.513263:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513270:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513276:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513281:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513286:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.513292:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4298598, setting rss=4298598; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.513721:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513729:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513736:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513741:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367cc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.513744:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513750:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.513752:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.513759:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e70eb lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.513773:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333963264:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016142.516229:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7600 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.516265:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1601536 1605631) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 5/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016142.516302:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7980000 - object 0x280000400:8 offset 1601536<>1605632 00000100:00100000:3.0:1734016142.516321:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016142.516333:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333963648:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016142.517950:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8800b6ee7780 00010000:00010000:3.0:1734016142.517982:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3534848 3538943) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e70d6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4890624->4898815) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271173f3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016142.518027:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981c00 - object 0x240000400:6 offset 3534848<>4194304 00000008:00010000:3.0:1734016142.543772:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### no dlm lock: extent: ffff8800b6ee7780 00000008:00010000:3.0:1734016142.543837:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### no dlm lock: extent: ffff8800b6ee7780 00000100:00100000:3.0:1734016142.543853:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333963904:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016142.580578:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333964288:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.585554:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335414528 00000100:00100000:3.0:1734016142.585558:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335414528 00000100:00100000:3.0:1734016142.585577:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12635 00000100:00100000:3.0:1734016142.585583:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335414400:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016142.585596:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e70d6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4890624->4898815) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271173f3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.585606:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e70d6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4890624->4898815) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271173f3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016142.585633:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 734, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016142.585635:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 176 00000020:00010000:3.0:1734016142.585638:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016142.585639:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 176 00000100:00100000:3.0:1734016142.585681:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335414400:12345-192.168.201.116@tcp:106: Request processed in 97us (343us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.585687:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12635 00000100:00100000:3.0:1734016142.585702:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12636 00000100:00100000:3.0:1734016142.585706:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335414528:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016142.585711:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.585717:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016142.585726:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 734, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016142.585727:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2342912 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000020:00010000:3.0:1734016142.585729:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016142.585730:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2342912 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000100:00100000:3.0:1734016142.585743:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335414528:12345-192.168.201.116@tcp:106: Request processed in 37us (344us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.585747:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12636 00010000:00010000:3.0:1734016142.589981:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.589993:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.590002:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.590007:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.590012:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.590019:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4298598, setting rss=4298598; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.590028:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.590032:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.590038:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.590043:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.590045:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.590052:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.590055:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.590061:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e70f9 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.590082:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333964288:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.594189:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.600877:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.611690:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.618242:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.626249:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.636637:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.651584:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.651664:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016142.660147:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8492902 i_blocks 280 00000020:00010000:3.0:1734016142.660154:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016142.660156:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.660161:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7630 (ffff8800a8a54908/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.660164:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7670 (ffff88012dc12098/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016142.660166:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b5a12480 (ffff88012bf7f3f8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.660168:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7630 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016142.660171:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7670 (ffff88012bf7f3f8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016142.660183:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016142.660202:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016142.660207:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016142.660215:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016142.660231:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.660234:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4c30, osc lock ffff8800a8a27300, flags 0x400001002 00000008:00010000:3.0:1734016142.660238:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016142.660240:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4c70, osc lock ffff8800a8a271d0, flags 0x400001002 00010000:00010000:3.0:1734016142.660258:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.660265:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.660318:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34b3c0/0xa50fd972cb8e7138 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.660324:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34b3c0/0xa50fd972cb8e7138 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.661547:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333965952:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.664548:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.664594:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016142.670354:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.670364:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.670371:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.670377:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.670382:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.670390:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4298598, setting rss=4298598; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.670406:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.670411:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.670417:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.670421:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa349d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.670424:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.670431:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.670435:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.670439:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa349d40/0xa50fd972cb8e7131 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.670490:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333965824:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.691570:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335416192 00000100:00100000:3.0:1734016142.691579:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335416192 00000100:00100000:3.0:1734016142.691624:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12648 00000100:00100000:3.0:1734016142.691630:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335416192:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016142.691645:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e711c lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3162112->3170303) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117439 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.691711:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016142.691717:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335416192:12345-192.168.201.116@tcp:104: Request processed in 87us (303us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.691723:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12648 00000100:00100000:3.0:1734016142.700544:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335416320 00000100:00100000:3.0:1734016142.700551:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335416320 00000100:00100000:3.0:1734016142.700593:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12649 00000100:00100000:3.0:1734016142.700599:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6465:x1818247335416320:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016142.700607:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e714d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [434176->438271] (req 434176->438271) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711746a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.700615:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e714d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [434176->438271] (req 434176->438271) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711746a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.700623:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e714d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->438271) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711746a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.700630:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e714d lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->438271) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711746a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.700637:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e714d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->438271) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711746a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.700645:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e714d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->438271) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711746a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.700674:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa365680) 00000100:00100000:3.0:1734016142.700713:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6465:x1818247335416320:12345-192.168.201.116@tcp:105: Request processed in 114us (247us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.700720:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12649 00000100:00100000:3.0:1734016142.727603:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016142.727619:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335416448 00000100:00100000:3.0:1734016142.727626:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335416448 00000100:00100000:3.0:1734016142.727671:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12650 00000100:00100000:3.0:1734016142.727678:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f858850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335416448:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016142.727693:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e714d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->438271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711746a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.727702:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e714d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->438271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711746a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016142.727723:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016142.727725:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 200 00000020:00010000:3.0:1734016142.727727:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016142.727729:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 200 00000100:00100000:3.0:1734016142.727767:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f858850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335416448:12345-192.168.201.116@tcp:106: Request processed in 90us (426us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.727774:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12650 00000100:00100000:3.0:1734016142.734803:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335416576 00000100:00100000:3.0:1734016142.734811:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335416576 00000100:00100000:3.0:1734016142.734855:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12651 00000100:00100000:3.0:1734016142.734862:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335416576:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016142.734874:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e714d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->438271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711746a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.734882:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e714d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->438271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711746a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016142.734905:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016142.734907:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 200 00000020:00010000:3.0:1734016142.734909:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016142.734910:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4298598 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 200 00000100:00100000:3.0:1734016142.734949:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335416576:12345-192.168.201.116@tcp:106: Request processed in 88us (235us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.734954:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12651 00000100:00100000:3.0:1734016142.743270:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335416704 00000100:00100000:3.0:1734016142.743278:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335416704 00000100:00100000:3.0:1734016142.743333:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12652 00000100:00100000:3.0:1734016142.743340:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335416704:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016142.743353:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa34a880/0xa50fd972cb8e7082 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1724416->1736703) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711739f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.743988:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016142.743995:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335416704:12345-192.168.201.116@tcp:104: Request processed in 655us (996us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.744002:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12652 00000100:00100000:3.0:1734016142.756793:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335416832 00000100:00100000:3.0:1734016142.756806:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335416832 00000100:00100000:3.0:1734016142.756864:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12653 00000100:00100000:3.0:1734016142.756872:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335416832:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016142.756887:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e7162 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3416064->3424255] (req 3416064->3424255) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711747f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.756897:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e7162 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3416064->3424255] (req 3416064->3424255) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711747f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.756906:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e7162 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3416064->3424255) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711747f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.756916:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e7162 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3416064->3424255) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711747f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.756925:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e7162 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3416064->3424255) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711747f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.756935:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e7162 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3416064->3424255) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711747f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.756970:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a781e1c0) 00000100:00100000:3.0:1734016142.757024:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335416832:12345-192.168.201.116@tcp:105: Request processed in 151us (2651us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.757030:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12653 00000100:00100000:3.0:1734016142.772098:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335416960 00000100:00100000:3.0:1734016142.772117:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335416960 00000100:00100000:3.0:1734016142.772530:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12654 00000100:00100000:3.0:1734016142.772536:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6478:x1818247335416960:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016142.772551:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e700b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117328 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016142.772889:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016142.772895:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6478:x1818247335416960:12345-192.168.201.116@tcp:104: Request processed in 359us (1475us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.772902:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12654 00000100:00100000:3.0:1734016142.785650:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.791685:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016142.791700:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335417216 00000100:00100000:3.0:1734016142.791707:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335417216 00000100:00100000:3.0:1734016142.791741:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12656 00000100:00100000:3.0:1734016142.791747:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2de050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335417216:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016142.791760:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f180/0xa50fd972cb8e7169 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc2711748d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.791769:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f180/0xa50fd972cb8e7169 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc2711748d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.791777:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f180/0xa50fd972cb8e7169 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc2711748d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.791786:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f180/0xa50fd972cb8e7169 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc2711748d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.791793:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f180/0xa50fd972cb8e7169 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc2711748d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.791807:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f180/0xa50fd972cb8e7169 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc2711748d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.791820:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a781f180) 00000100:00100000:3.0:1734016142.791857:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2de050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335417216:12345-192.168.201.116@tcp:105: Request processed in 110us (271us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.791865:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12656 00000100:00100000:3.0:1734016142.799255:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.811387:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.879214:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016142.879228:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335418880 00000100:00100000:3.0:1734016142.879236:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335418880 00000100:00100000:3.0:1734016142.879268:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12663 00000100:00100000:3.0:1734016142.879274:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dd050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335418880:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016142.879287:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781f180/0xa50fd972cb8e7169 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711748d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.879337:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016142.879342:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dd050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335418880:12345-192.168.201.116@tcp:104: Request processed in 68us (245us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.879348:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12663 00000100:00100000:3.0:1734016142.889333:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335419136 00000100:00100000:3.0:1734016142.889340:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335419136 00000100:00100000:3.0:1734016142.889369:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12664 00000100:00100000:3.0:1734016142.889374:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335419136:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016142.889386:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [303104->307199] (req 303104->307199) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.889394:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [303104->307199] (req 303104->307199) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.889401:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 303104->307199) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.889408:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 303104->307199) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.889414:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 303104->307199) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.889421:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 303104->307199) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.892337:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c21c0) 00000100:00100000:3.0:1734016142.892435:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335419136:12345-192.168.201.116@tcp:105: Request processed in 3051us (3343us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.892446:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12664 00000100:00100000:3.0:1734016142.912701:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335419264 00000100:00100000:3.0:1734016142.912709:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335419264 00000100:00100000:3.0:1734016142.912763:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12665 00000100:00100000:3.0:1734016142.912771:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335419264:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016142.912788:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 303104->307199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.912796:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 303104->307199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016142.912818:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016142.912820:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 168 00000020:00010000:3.0:1734016142.912824:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016142.912825:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 168 00000100:00100000:3.0:1734016142.912865:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335419264:12345-192.168.201.116@tcp:106: Request processed in 95us (240us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.912871:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12665 00010000:00010000:3.0:1734016142.915748:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.915765:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.915778:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.915785:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.915790:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.915804:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.915825:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.915831:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.915837:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.915842:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.915847:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.915856:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.915861:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.915866:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e71a8 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.916143:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333970304:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.927694:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335419392 00000100:00100000:3.0:1734016142.927702:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335419392 00000100:00100000:3.0:1734016142.927743:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12666 00000100:00100000:3.0:1734016142.927749:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335419392:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016142.927763:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 303104->307199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.927777:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 303104->307199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016142.927800:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016142.927802:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 168 00000020:00010000:3.0:1734016142.927806:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016142.927807:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 168 00000100:00100000:3.0:1734016142.927851:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335419392:12345-192.168.201.116@tcp:106: Request processed in 102us (402us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.927858:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12666 00010000:00010000:3.0:1734016142.937348:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.937366:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.937378:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.937384:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.937390:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016142.937405:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016142.938474:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.938484:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.938491:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.938497:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016142.938502:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.938513:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016142.938518:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.938524:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e71af lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016142.940688:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333970432:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016142.944561:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335419520 00000100:00100000:3.0:1734016142.944571:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335419520 00000100:00100000:3.0:1734016142.944618:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12667 00000100:00100000:3.0:1734016142.944626:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:8c598caf-7141-4c19-8566-0b7731516db9+5:13956:x1818247335419520:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016142.944640:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e719a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc271174cc expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016142.944710:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016142.944716:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:8c598caf-7141-4c19-8566-0b7731516db9+5:13956:x1818247335419520:12345-192.168.201.116@tcp:104: Request processed in 91us (297us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.944722:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12667 00000100:00100000:3.0:1734016142.962239:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.962532:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12668 00000100:00100000:3.0:1734016142.962539:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7982d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335419648:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016142.962553:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cd80/0xa50fd972cb8e7170 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117494 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.962617:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016142.962623:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7982d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335419648:12345-192.168.201.116@tcp:104: Request processed in 84us (389us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.962629:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12668 00000100:00100000:3.0:1734016142.968707:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016142.997393:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335420288 00000100:00100000:3.0:1734016142.997399:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335420288 00000100:00100000:3.0:1734016142.997689:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12673 00000100:00100000:3.0:1734016142.997695:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335420288:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016142.997703:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 303104->307199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016142.997709:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e71a1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 303104->307199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174d3 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016142.997725:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016142.997726:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 1902126 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 136 00000020:00010000:3.0:1734016142.997729:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016142.997730:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 1902126 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 136 00000100:00100000:3.0:1734016142.997747:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335420288:12345-192.168.201.116@tcp:106: Request processed in 53us (375us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016142.997752:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12673 00000100:00100000:3.0:1734016143.005883:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335420416 00000100:00100000:3.0:1734016143.005892:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335420416 00000100:00100000:3.0:1734016143.010318:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335420544 00000100:00100000:3.0:1734016143.010327:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335420544 00000100:00100000:3.0:1734016143.010370:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12675 00000100:00100000:3.0:1734016143.010376:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335420544:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016143.010389:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e71e0 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1888256->1900543] (req 1888256->1900543) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117520 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.010398:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e71e0 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1888256->1900543] (req 1888256->1900543) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117520 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.010404:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e71e0 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1888256->1900543) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117520 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.010415:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e71e0 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1888256->1900543) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27117520 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.010423:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e71e0 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1888256->1900543) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27117520 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.010432:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e71e0 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1888256->1900543) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc27117520 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.010448:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366f40) 00000100:00100000:3.0:1734016143.010513:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7698:x1818247335420544:12345-192.168.201.116@tcp:105: Request processed in 137us (513us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.010520:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12675 00000100:00100000:3.0:1734016143.060914:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.070789:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.079418:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.079454:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335421056 00000100:00100000:3.0:1734016143.079463:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335421056 00000100:00100000:3.0:1734016143.079521:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12678 00000100:00100000:3.0:1734016143.079529:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335421056:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016143.079546:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c6c0/0xa50fd972cb8e71ee lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117535 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.079612:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016143.079618:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335421056:12345-192.168.201.116@tcp:104: Request processed in 91us (209us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.079624:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12678 00000100:00100000:3.0:1734016143.079666:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.079677:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335421184 00000100:00100000:3.0:1734016143.079680:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335421184 00000100:00100000:3.0:1734016143.079695:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12679 00000100:00100000:3.0:1734016143.079699:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335421184:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016143.079706:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e71e0 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1888256->1900543) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117520 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.079730:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016143.079734:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335421184:12345-192.168.201.116@tcp:104: Request processed in 34us (68us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.079738:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12679 00000100:00100000:3.0:1734016143.093935:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335421312 00000100:00100000:3.0:1734016143.093945:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335421312 00000100:00100000:3.0:1734016143.094125:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12680 00000100:00100000:3.0:1734016143.094131:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7698:x1818247335421312:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016143.094144:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7203 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1396736->1400831] (req 1396736->1400831) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711754a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.094154:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7203 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1396736->1400831] (req 1396736->1400831) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711754a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.094161:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7203 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1396736->1400831) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711754a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.094173:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7203 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1396736->1400831) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711754a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.094180:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7203 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1396736->1400831) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711754a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.094190:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7203 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1396736->1400831) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711754a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.094340:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa367cc0) 00000100:00100000:3.0:1734016143.094976:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7698:x1818247335421312:12345-192.168.201.116@tcp:105: Request processed in 843us (2437us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.094984:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12680 00000100:00100000:3.0:1734016143.104734:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335421440 00000100:00100000:3.0:1734016143.104742:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335421440 00000100:00100000:3.0:1734016143.104780:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12681 00000100:00100000:3.0:1734016143.104788:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335421440:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.104801:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.104810:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.104833:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.104835:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 0 00000020:00010000:3.0:1734016143.104838:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016143.104839:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 0 00000100:00100000:3.0:1734016143.104880:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335421440:12345-192.168.201.116@tcp:106: Request processed in 92us (319us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.104885:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12681 00000100:00100000:3.0:1734016143.110899:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335421568 00000100:00100000:3.0:1734016143.110906:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335421568 00000100:00100000:3.0:1734016143.110944:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12682 00000100:00100000:3.0:1734016143.110950:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335421568:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.110965:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.110974:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.110995:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.110997:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 0 00000020:00010000:3.0:1734016143.111000:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016143.111001:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 0 00000100:00100000:3.0:1734016143.111041:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335421568:12345-192.168.201.116@tcp:106: Request processed in 90us (229us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.111047:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12682 00010000:00010000:3.0:1734016143.113618:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.113633:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.113644:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.113650:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.113656:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.113673:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016143.113683:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.113689:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.113695:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.113700:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016143.113704:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.113713:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016143.113717:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.113722:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7211 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.114142:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333973760:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.118690:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016143.118721:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335421696 00000100:00100000:3.0:1734016143.118729:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335421696 00000100:00100000:3.0:1734016143.118973:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12683 00000100:00100000:3.0:1734016143.118981:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f858050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335421696:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.118997:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.119006:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.119030:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.119032:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 0 00000020:00010000:3.0:1734016143.119035:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016143.119036:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 0 00000100:00100000:3.0:1734016143.119827:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f858050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335421696:12345-192.168.201.116@tcp:106: Request processed in 843us (2386us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.119836:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12683 00000008:00010000:3.0:1734016143.162847:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7203 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1396736->1400831) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711754a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.163017:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7203 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1396736->1400831) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711754a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.163042:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333974144:192.168.201.116@tcp:3:fsx.0 00010000:00010000:3.0:1734016143.168946:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.168960:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.168975:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.168981:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.168987:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.169000:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016143.169009:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.169014:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.169021:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.169027:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016143.169033:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.169042:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016143.169047:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.169057:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e721f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.170202:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333974272:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.172705:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.183082:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.212178:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.228444:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.246579:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.262598:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.277933:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.294864:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.294920:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016143.322903:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10243558 i_blocks 160 00000020:00010000:3.0:1734016143.322909:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016143.322912:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.322916:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7630 (ffff88012dc12b48/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.322919:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7670 (ffff88012dc126c8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.322921:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.322923:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7630 (ffff88012dc127e8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.322927:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7670 (ffff88012dc127e8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016143.322938:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016143.322947:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10243558 00000080:00200000:3.0:1734016143.323000:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016143.323004:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016143.323016:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016143.323028:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [222, 224] 00000008:00010000:3.0:1734016143.323040:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016143.323043:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58ba330, osc lock ffff8800a8a26d10, flags 0x0 00010000:00010000:3.0:1734016143.323057:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.323066:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (909312 921599) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.323074:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.323080:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.323085:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016143.323094:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10243558 i_blocks 160 00000020:00010000:3.0:1734016143.327938:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b63204e8 (ffff88012dc127e8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016143.327942:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63204e8 (ffff88012dc127e8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.327947:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b58ba330 (ffff88012dc126c8/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016143.327957:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.327966:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.327984:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63204e8 (ffff88012dc127e8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.327986:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b58ba330 (ffff88012dc127e8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016143.327999:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016143.328020:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016143.328027:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016143.328043:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016143.328056:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016143.328064:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016143.328074:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:3.0:1734016143.328076:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e71e7 00000080:00010000:3.0:1734016143.328080:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e71e7 from it ffff8801369fbcd8 00010000:00010000:3.0:1734016143.328083:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016143.328088:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016143.328114:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016143.328122:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016143.328137:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016143.328140:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f30, osc lock ffff8800a8a26d10, flags 0x400001002 00000008:00010000:3.0:1734016143.328145:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016143.328146:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f70, osc lock ffff8800a8a26e40, flags 0x400001002 00010000:00010000:3.0:1734016143.328158:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328165:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328174:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328180:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328193:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328199:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328206:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328211:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016143.328227:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10243558 i_blocks 152 00000020:00010000:3.0:1734016143.328230:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016143.328232:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.328234:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f30 (ffff88012dc126c8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.328237:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f70 (ffff88012dc12b48/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.328239:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.328241:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f30 (ffff88012dc127e8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.328243:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f70 (ffff88012dc127e8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016143.328249:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016143.328258:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016143.328262:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016143.328265:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016143.328269:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016143.328270:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f30, osc lock ffff8800a8a26e40, flags 0x400001002 00000008:00010000:3.0:1734016143.328272:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016143.328273:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f70, osc lock ffff8800a8a26d10, flags 0x400001002 00010000:00010000:3.0:1734016143.328278:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328285:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328291:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328296:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328304:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328314:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328320:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.328326:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016143.328333:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10243558 i_blocks 152 00000020:00010000:3.0:1734016143.328335:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016143.328337:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.328338:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f30 (ffff88012dc12b48/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.328340:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f70 (ffff88012dc126c8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.328342:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.328344:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f30 (ffff88012dc127e8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.328346:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f70 (ffff88012dc127e8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016143.328350:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016143.328356:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10243558 00000080:00200008:3.0:1734016143.328363:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016143.328367:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016143.328372:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016143.328377:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016143.328386:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016143.329589:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:3.0:1734016143.329596:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e71e7 00000080:00010000:3.0:1734016143.329604:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e71e7 from it ffff8801369fbcd8 00010000:00010000:3.0:1734016143.329607:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016143.329615:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016143.329636:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016143.329645:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016143.329660:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016143.329662:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f30, osc lock ffff8800a8a26d10, flags 0x400001002 00000008:00010000:3.0:1734016143.329667:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016143.329668:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f70, osc lock ffff8800a8a26e40, flags 0x400001002 00010000:00010000:3.0:1734016143.329679:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.329688:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.329695:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.329701:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.329712:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.329719:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.329724:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.329730:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016143.329742:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10243558 i_blocks 152 00000020:00010000:3.0:1734016143.329746:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016143.329749:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.329751:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f30 (ffff88012dc126c8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.329754:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f70 (ffff88012dc12b48/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.329756:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.329757:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f30 (ffff88012dc127e8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.332027:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f70 (ffff88012dc127e8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016143.332057:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016143.332087:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016143.332096:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016143.332115:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016143.332137:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016143.332141:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f30, osc lock ffff8800a8a26e40, flags 0x400001002 00000008:00010000:3.0:1734016143.332146:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016143.332147:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f70, osc lock ffff8800a8a26d10, flags 0x400001002 00010000:00010000:3.0:1734016143.332169:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.332183:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.332194:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.332203:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.332224:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.332231:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.332238:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.332243:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016143.332258:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10243558 i_blocks 152 00000020:00010000:3.0:1734016143.332262:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016143.332264:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.332266:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f30 (ffff88012dc12b48/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.332269:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f70 (ffff88012dc126c8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.332271:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbeeb58 (ffff88012dc127e8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.332273:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f30 (ffff88012dc127e8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.332275:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f70 (ffff88012dc127e8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016143.332279:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016143.332286:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10243558 00000080:00200008:3.0:1734016143.332365:0:10406:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8) current size 10243558, valid attrs 2068, mode 8801, uid -1227447776, gid -30720, new size 3472655, atime 1734016143.000000000, mtime 1734016143.000000000, ctime 1734016143.000000000 00010000:00010000:3.0:1734016143.332394:0:10406:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016143.332410:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781c900/0xa50fd972cb8e5f33 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115fb0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016143.332451:0:10406:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016143.332458:0:10406:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1734016143.332460:0:10406:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c1440/0xa50fd972cb8e71e7 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc2711752e expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000020:00100000:3.0:1734016143.333012:0:10406:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff88012a5ff800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1734016143.333023:0:10406:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a7983480 x1818247333976576/t0(0) o36->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016143.333042:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333976576:192.168.201.116@tcp:36:fsx.0 00000100:00100000:3.0:1734016143.333110:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4f00 going to sleep for 11 seconds 00000100:00100000:3.0:1734016143.338537:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.346610:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333976576:192.168.201.116@tcp:36:fsx.0 00000080:00200000:3.0:1734016143.346656:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016143.346661:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016143.346687:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016143.346708:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1734016143.346722:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016143.346727:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f30, osc lock ffff8800a8a26d10, flags 0x0 00000008:00010000:3.0:1734016143.346730:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016143.346731:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f70, osc lock ffff8800a8a26e40, flags 0x0 00010000:00010000:3.0:1734016143.346749:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.346761:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.346772:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.346779:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.346784:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.346794:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.346805:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.346811:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.346816:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.346821:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.346894:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016143.346908:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333976832:192.168.201.116@tcp:10:fsx.0 00000100:00100000:3.0:1734016143.348479:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333976960:192.168.201.116@tcp:10:fsx.0 00000080:00200000:3.0:1734016143.361251:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 3472655 i_blocks 144 00000020:00010000:3.0:1734016143.361259:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b63204e8 (ffff88012dc127e8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016143.361263:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63204e8 (ffff88012dc127e8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.361268:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f30 (ffff88012dc126c8/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016143.361277:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.361288:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.361308:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f70 (ffff88012dc12b48/1) at cl_lock_cancel():150 00010000:00010000:3.0:1734016143.361312:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.361318:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.361326:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63204e8 (ffff88012dc127e8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.361328:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f30 (ffff88012dc127e8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.361333:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f70 (ffff88012dc127e8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016143.361349:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016143.361362:0:10406:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8) current size 3472655, valid attrs 2168, mode 8801, uid -1227447776, gid -30720, new size 3472655, atime 1734016143.000000000, mtime 1734016143.331000000, ctime 1734016143.331000000, rc 0 00000080:00200008:3.0:1734016143.361386:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016143.361392:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016143.361406:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016143.361418:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:3.0:1734016143.361456:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e726c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:3.0:1734016143.361461:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e726c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:3.0:1734016143.361510:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e726c lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:3.0:1734016143.361528:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333977088:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.361591:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4f00 going to sleep for 11 seconds 00000008:00010000:3.0:1734016143.394543:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.394595:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3821568 3825663) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016143.394629:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982d80 - object 0x240000400:6 offset 3821568<>3825664 00000008:00010000:3.0:1734016143.401297:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.401544:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7a80 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.401565:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333977856:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016143.408358:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7780 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.408390:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7780 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.408408:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333978112:192.168.201.116@tcp:3:fsx.0 00010000:00010000:0.0:1734016143.408598:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408611:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016143.408624:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6320b78 (ffff88012dc12ea8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.408626:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811830 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.408639:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016143.408650:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016143.408661:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016143.408670:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [934, 934] 00000008:00010000:0.0:1734016143.408679:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016143.408683:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999cb0, osc lock ffff8800a9f23300, flags 0x0 00010000:00010000:0.0:1734016143.408697:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408707:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3825664 3829759) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408716:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408722:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408728:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000001:00010000:0.0:1734016143.408746:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016143.408750:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.408751:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55ab0, osc lock ffff8800a9f22d10, flags 0x400001002 00000008:00010000:0.0:1734016143.408758:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.408759:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55af0, osc lock ffff8800a9f224c0, flags 0x400001002 00010000:00010000:0.0:1734016143.408763:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408770:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408776:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 8/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408781:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 7/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408789:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408795:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408801:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408806:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e71b6 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271174ef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016143.408819:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 3827071 i_blocks 144 00000020:00010000:0.0:1734016143.408822:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.408824:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.408825:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55ab0 (ffff88012dc12638/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.408827:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55af0 (ffff8800a8a54908/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.408829:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.408830:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55ab0 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.408833:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55af0 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000008:00010000:0.0:1734016143.408856:0:10406:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800a8a55a80 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.408862:0:10406:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55a80 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.408867:0:10406:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.408874:0:10406:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016143.408884:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134a48, count 1, dirtied 1 00000020:00010000:0.0:1734016143.408892:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b6320b78 (ffff88012dc12ea8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.408894:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6320b78 (ffff88012dc12ea8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.408896:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999cb0 (ffff88012dc12638/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016143.408899:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.408908:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016143.408914:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6320b78 (ffff88012dc12ea8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.408915:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999cb0 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.408922:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016143.408941:0:10406:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8), start 3821568, end 3829759, datasync 1 00000080:00200000:0.0:1734016143.408946:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016143.408949:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1734016143.408958:0:10406:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.408967:0:10406:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.412515:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3821568 3825663) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016143.412587:0:10406:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a934ca80 - object 0x240000400:6 offset 3821568<>3827071 00000100:00100000:0.0:1734016143.412655:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016143.412674:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333978240:192.168.201.116@tcp:4:fsx.0 00000008:00010000:0.0:1734016143.416504:0:1982:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.417639:0:1982:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.417672:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333978240:192.168.201.116@tcp:4:fsx.0 00000100:00100000:3.0:1734016143.424738:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333978496:192.168.201.116@tcp:16:fsx.0 00000080:00200000:0.0:1734016143.437656:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 3827071 i_blocks 160 00000080:00200000:0.0:1734016143.437671:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016143.437781:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016143.437788:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016143.437807:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016143.437820:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016143.437866:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016143.437873:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016143.437881:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016143.437898:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333978624:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.437966:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55300 going to sleep for 11 seconds 00000100:00100000:0.0:1734016143.448251:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333978624:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.448272:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.448282:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.448292:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.448300:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.448308:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.448313:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.448318:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016143.448325:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a934ea00 x1818247333978624/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016159 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016143.448333:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016143.448341:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.448351:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016143.448396:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016143.448400:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e7273 00000080:00010000:0.0:1734016143.448404:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016143.448407:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7273 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016143.448409:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.448418:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e7273 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271175d6 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016143.448446:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016143.448455:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016143.448506:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.448510:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55330, osc lock ffff8800a9f23300, flags 0x400001002 00000008:00010000:0.0:1734016143.448515:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.448517:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55370, osc lock ffff8800a9f224c0, flags 0x400001002 00010000:00010000:0.0:1734016143.448536:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.448544:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.448602:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.448608:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.448646:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333978880:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.448709:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016143.448716:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333978752:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.452148:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.457639:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016143.465491:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465514:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465525:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465531:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465537:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.465551:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3827071, setting rss=3827071; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.465558:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465563:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465569:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465575:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c33c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.465580:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465596:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.465602:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465606:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e727a lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.465647:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333978752:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.465662:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465667:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465673:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465677:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465682:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.465689:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.465707:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465713:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465718:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465721:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.465724:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465729:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.465731:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.465736:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e7281 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.465750:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333978880:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.468170:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333979008:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.468251:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016143.468257:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333979136:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.473792:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.474679:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016143.479700:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.479722:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.479732:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.479737:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.479741:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.479754:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3827071, setting rss=3827071; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.479760:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.479765:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.479772:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.479777:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781fa80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.479783:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.479800:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.479805:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.479810:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e7288 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.479852:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333979008:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.480319:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.480326:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.480333:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.480337:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.480342:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.480348:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.480366:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.480372:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.480379:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.480383:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781e1c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.480387:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.480394:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.480399:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.480403:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781e1c0/0xa50fd972cb8e728f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.480415:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333979136:192.168.201.116@tcp:101:fsx.0 00000008:00010000:0.0:1734016143.488123:0:1982:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a7b19600 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.488178:0:1982:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3567616 3571711) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016143.488312:0:1982:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a934ce00 - object 0x240000400:6 offset 0<>3821568 00000100:00100000:0.0:1734016143.488863:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016143.488875:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333979392:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016143.569151:0:1982:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a7b19600 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.569485:0:1982:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a7b19600 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.569512:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333979392:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016143.584400:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.584454:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016143.604517:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.604534:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.604548:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.604555:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.604560:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.604572:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016143.604582:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.604587:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.604592:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.604602:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016143.604607:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.604617:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016143.604622:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.604627:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e72a4 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.604669:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333979776:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016143.620643:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620662:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620674:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620681:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620686:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.620700:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3827071, setting rss=3827071; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016143.620707:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620713:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620719:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620724:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016143.620729:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620739:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016143.620744:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620750:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72ab lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.620779:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333979904:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016143.620794:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620804:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620811:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620816:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620822:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.620827:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016143.620833:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620838:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620843:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620848:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016143.620851:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620860:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016143.620862:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.620867:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b2 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.620879:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333980032:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.634264:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016143.634284:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335424512 00000100:00100000:3.0:1734016143.634292:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335424512 00000100:00100000:3.0:1734016143.634324:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12704 00000100:00100000:3.0:1734016143.634331:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f858850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6479:x1818247335424512:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016143.634345:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7296 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117600 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016143.634412:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016143.634418:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f858850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6479:x1818247335424512:12345-192.168.201.116@tcp:104: Request processed in 87us (296us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.634424:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12704 00000100:00100000:3.0:1734016143.662713:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335424640 00000100:00100000:3.0:1734016143.662721:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335424640 00000100:00100000:3.0:1734016143.662760:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12705 00000100:00100000:3.0:1734016143.662767:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335424640:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016143.662780:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.662836:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016143.662841:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335424640:12345-192.168.201.116@tcp:104: Request processed in 74us (229us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.662847:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12705 00010000:00010000:0.0:1734016143.666056:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016143.666069:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.666078:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.666084:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016143.666115:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016143.669327:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016143.669378:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.669386:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016143.669411:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.669418:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7249 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6045696->6049791) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117590 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.669437:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016143.669439:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016143.669442:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016143.669452:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016143.669490:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333980672:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016143.684355:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016143.688751:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333980672:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016143.694552:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016143.701492:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333980800:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.701523:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.701533:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.701540:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.702275:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335425024 00000100:00100000:3.0:1734016143.702281:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335425024 00000100:00100000:3.0:1734016143.702320:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12708 00000100:00100000:3.0:1734016143.702327:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335425024:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016143.702336:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.702344:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.702352:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.702359:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.702366:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.702374:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.702396:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa365f80) 00000100:00100000:3.0:1734016143.702436:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335425024:12345-192.168.201.116@tcp:105: Request processed in 109us (1081us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.702441:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12708 00010000:00010000:0.0:1734016143.702550:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.702558:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.702576:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.702593:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.702604:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.702610:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.702701:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333981184:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016143.702736:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016143.702743:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333981056:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016143.707137:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333981056:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016143.710291:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333981184:192.168.201.116@tcp:10:fsx.0 00000100:00100000:3.0:1734016143.726390:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.726530:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.726561:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016143.726579:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335425280 00000100:00100000:3.0:1734016143.726587:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335425280 00000100:00100000:3.0:1734016143.726616:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335425152 00000100:00100000:3.0:1734016143.726617:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335425152 00000100:00100000:3.0:1734016143.726630:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12709 00000100:00100000:3.0:1734016143.726637:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335425280:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.726651:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.726661:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.726684:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.726686:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000020:00010000:3.0:1734016143.726689:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016143.726690:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000100:00100000:3.0:1734016143.726723:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335425280:12345-192.168.201.116@tcp:106: Request processed in 87us (344us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.726729:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12709 00000100:00100000:3.0:1734016143.726739:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12710 00000100:00100000:3.0:1734016143.726743:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2ddc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335425152:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.726748:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.726754:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.726765:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.726766:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 0 00000020:00010000:3.0:1734016143.726769:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016143.726770:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 0 00000100:00100000:3.0:1734016143.726782:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2ddc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335425152:12345-192.168.201.116@tcp:106: Request processed in 39us (254us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.726787:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12710 00000080:00200000:3.0:1734016143.755491:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6668192 i_blocks 160 00000020:00010000:3.0:1734016143.755497:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016143.755500:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.755505:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7930 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.755508:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7970 (ffff8800a8a54908/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016143.755510:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.755511:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7930 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016143.755514:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7970 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016143.755525:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016143.755542:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016143.755546:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016143.755555:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016143.755566:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016143.755568:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f30, osc lock ffff8800a8a26e40, flags 0x400001002 00000008:00010000:3.0:1734016143.755572:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016143.755573:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4f70, osc lock ffff8800a8a26d10, flags 0x400001002 00010000:00010000:3.0:1734016143.755590:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.755597:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.755644:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e72e3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.755649:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a880/0xa50fd972cb8e72e3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.758708:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.758794:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016143.762288:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.762301:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.762309:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.762314:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.762323:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.762331:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016143.762349:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.762355:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.762361:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.762366:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa348b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016143.762370:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.762376:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016143.762379:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.762384:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348b40/0xa50fd972cb8e72dc lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.762415:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333981696:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016143.762507:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6668192 i_blocks 160 00000020:00010000:0.0:1734016143.762516:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.762518:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.762524:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f30 (ffff8800a8a54908/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.762526:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d4f70 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.762528:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.762530:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f30 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.762534:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d4f70 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.762545:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016143.762555:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 6668192 00000080:00200008:0.0:1734016143.762570:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016143.762575:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016143.762590:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016143.762601:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016143.762632:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016143.762638:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016143.762646:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016143.762662:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333981952:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.762701:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1734016143.765487:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333981952:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.765507:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.765516:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.765528:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.765536:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.765549:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.765556:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.765562:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016143.765569:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a934d180 x1818247333981952/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016159 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016143.765578:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016143.765587:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.765597:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016143.765650:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016143.765654:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e72ea 00000080:00010000:0.0:1734016143.765658:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016143.765661:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e72ea from it ffff8801369fbcd8 00010000:00010000:0.0:1734016143.765663:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.765669:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016143.765700:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016143.765710:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016143.765729:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.765733:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f22d10, flags 0x400001002 00000008:00010000:0.0:1734016143.765738:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.765740:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23430, flags 0x400001002 00010000:00010000:0.0:1734016143.765751:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765760:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765770:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765776:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765790:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765796:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765802:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765812:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016143.765824:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6668192 i_blocks 160 00000020:00010000:0.0:1734016143.765827:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.765830:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.765831:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.765834:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff8800a8a54908/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.765835:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.765837:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.765839:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.765846:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016143.765861:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016143.765864:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016143.765867:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016143.765871:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.765872:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23430, flags 0x400001002 00000008:00010000:0.0:1734016143.765875:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.765876:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f22d10, flags 0x400001002 00010000:00010000:0.0:1734016143.765881:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765887:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765894:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765900:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765909:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765914:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765920:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.765924:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016143.765931:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6668192 i_blocks 160 00000020:00010000:0.0:1734016143.765933:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.765934:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.765936:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff8800a8a54908/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.765937:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.765939:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.765940:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.765942:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.765945:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016143.765950:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 6668192 00000080:00200008:0.0:1734016143.766639:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 8981321, type: SEEK_SET 00000080:00200008:0.0:1734016143.766645:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 8981321, type: SEEK_SET, rc = 8981321 00000080:00200008:0.0:1734016143.766653:0:10406:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 8981321, count: 1614 00000080:00200000:0.0:1734016143.766660:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 8981321, bytes: 1614 00000080:00200000:0.0:1734016143.766663:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016143.766668:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016143.766669:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [8981321, 8982935) 0 0 00000080:00200000:0.0:1734016143.766671:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016143.766679:0:10406:0:(file.c:1893:ll_file_io_generic()) Range lock [2192, 2193] 00020000:00200000:0.0:1734016143.766685:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 2 chunk: [8981321, 8982935) 8982935, 1614 00020000:00200000:0.0:1734016143.766689:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016143.766701:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2192, 2193] 00000008:00010000:0.0:1734016143.766708:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016143.766710:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999cb0, osc lock ffff8800a9f22d10, flags 0x0 00010000:00010000:0.0:1734016143.766717:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.766724:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4784128 4792319) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.766730:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.766736:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.766741:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016143.766748:0:10406:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [8981321, 8982935) 00000080:00200000:0.0:1734016143.766770:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 2192 of 2889 to 1207 bytes 00000080:00200000:0.0:1734016143.766801:0:10406:0:(rw26.c:1006:ll_write_end()) pos 8981321, len 1207, copied 1207 00000080:00200000:0.0:1734016143.766806:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a78ce948[3 ffff8800b3a5cdc8 2 1 ffff88012dbef968] 00000080:00200000:0.0:1734016143.766808:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a57ec0 1fffff00000805 3:0 ffff8800a78ce948 2192 lru 00000080:00200000:0.0:1734016143.766815:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a78ce9a8 1168: 1< 0 - - > 2< 4784128 0 0 0x0 0x0 | ffff88012a41dcc8 ffff8800a9006000 > 3< 0 0 > 4< 0 0 8 489881600 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016143.766816:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a78ce948 00000080:00200000:0.0:1734016143.766817:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:0.0:1734016143.766819:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 2193 of 0 to 407 bytes 00000080:00200000:0.0:1734016143.766830:0:10406:0:(rw26.c:1006:ll_write_end()) pos 8982528, len 407, copied 407 00000080:00200000:0.0:1734016143.766833:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a78cea20[3 ffff8800b3a5cdc8 2 1 ffff88012dbef968] 00000080:00200000:0.0:1734016143.766836:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a57f00 1fffff00000805 3:0 ffff8800a78cea20 2193 lru 00000080:00200000:0.0:1734016143.766840:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a78cea80 1169: 1< 0 - - > 2< 4788224 0 0 0x0 0x0 | ffff88012a41dcc8 ffff8800a9006000 > 3< 0 0 > 4< 0 0 8 489881600 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016143.766842:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a78cea20 00000080:00200000:0.0:1734016143.766842:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:0.0:1734016143.766845:0:10406:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 2, from 2889, to 407 00000008:00010000:0.0:1734016143.766866:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800a8a55cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.766885:0:10406:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800a8a55cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016143.766893:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134248, count 2, dirtied 2 00000080:00200000:0.0:1734016143.766898:0:10406:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 2 pages 1614 bytes, tot: 1614 00000080:00200000:0.0:1734016143.766901:0:10406:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 1614 00000020:00010000:0.0:1734016143.766911:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b6320b78 (ffff88012dc12ea8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.766914:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6320b78 (ffff88012dc12ea8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.766916:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999cb0 (ffff88012dc12368/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016143.766919:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.766929:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016143.766938:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6320b78 (ffff88012dc12ea8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.766940:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999cb0 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.766945:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 1614 bytes 00000080:00200000:0.0:1734016143.766948:0:10406:0:(file.c:1930:ll_file_io_generic()) Range unlock [2192, 2193] 00000080:00200000:0.0:1734016143.766952:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016143.766954:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 1614, restart: 0 00000080:00200000:0.0:1734016143.766960:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8982935 i_blocks 160 00000080:00200000:0.0:1734016143.766962:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 1614 00000080:00200008:0.0:1734016143.766973:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016143.766978:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016143.766989:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016143.766999:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e72c7 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711763f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.767005:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e72c7 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711763f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016143.767013:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016143.767016:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e72c7 00000080:00010000:0.0:1734016143.767019:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e72c7 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016143.767021:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e72c7 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc2711763f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.767026:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e72c7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711763f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016143.767037:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016143.767041:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016143.767047:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.767048:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55e70, osc lock ffff8800a9f22d10, flags 0x400001002 00000008:00010000:0.0:1734016143.767052:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.767053:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55eb0, osc lock ffff8800a9f23430, flags 0x400001002 00010000:00010000:0.0:1734016143.767069:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.767076:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.767142:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.767148:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.767179:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333982208:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.767250:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016143.767255:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333982080:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.773014:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773029:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773044:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773049:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773054:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.773065:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4788631, setting rss=4788631; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.773071:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773076:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773081:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773087:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.773100:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773109:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.773113:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773123:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e72f1 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.773156:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333982080:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.773169:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773175:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773180:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773185:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773189:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.773194:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2473888, setting rss=2473888; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.773201:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773205:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773214:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773218:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c33c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.773220:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773225:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.773227:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.773232:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e72f8 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.773242:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333982208:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.776190:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335425920 00000100:00100000:3.0:1734016143.776198:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335425920 00000100:00100000:3.0:1734016143.776229:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12715 00000100:00100000:3.0:1734016143.776235:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335425920:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.776246:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.776255:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.776278:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.776280:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4788631 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000020:00010000:3.0:1734016143.776283:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016143.776284:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4788631 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000100:00100000:3.0:1734016143.776321:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335425920:12345-192.168.201.116@tcp:106: Request processed in 86us (206us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.776326:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12715 00000100:00100000:3.0:1734016143.777537:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335426048 00000100:00100000:3.0:1734016143.777542:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335426048 00000100:00100000:3.0:1734016143.777561:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12716 00000100:00100000:3.0:1734016143.777565:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335426048:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.777573:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.777581:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.777604:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.777606:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000020:00010000:3.0:1734016143.777609:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016143.777610:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000100:00100000:3.0:1734016143.777632:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335426048:12345-192.168.201.116@tcp:106: Request processed in 67us (173us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.777637:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12716 00010000:00010000:3.0:1734016143.779018:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.779031:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.779043:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.779049:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.779054:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.779066:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4788631, setting rss=4788631; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016143.779074:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.779083:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.779089:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.779103:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016143.779107:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.779116:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016143.779121:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.779125:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e72ff lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.779152:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333982336:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016143.784187:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8982935 i_blocks 161 00000020:00010000:0.0:1734016143.784193:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.784195:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.784200:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7630 (ffff8800a8a54908/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.784203:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7670 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.784205:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbef908 (ffff88012dc12ea8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.784207:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7630 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.784211:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7670 (ffff88012dc12ea8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.784224:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016143.784234:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8982935 00000080:00200008:0.0:1734016143.784249:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016143.784254:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016143.784270:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016143.784562:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e72c7 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711763f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000100:00100000:0.0:1734016143.785338:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333982592:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.790275:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.791262:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016143.791366:0:9611:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1734016143.791382:0:9611:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335426304 00000100:00100000:0.0:1734016143.791389:0:9611:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335426304 00000100:00100000:0.0:1734016143.791427:0:9611:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 230 00000100:00100000:0.0:1734016143.791432:0:9611:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89fc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8216:x1818247335426304:12345-192.168.201.116@tcp:106: 00010000:00010000:0.0:1734016143.791445:0:9611:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.791455:0:9611:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016143.791493:0:9611:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:0.0:1734016143.791495:0:9611:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000020:00010000:0.0:1734016143.791498:0:9611:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:0.0:1734016143.791499:0:9611:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000100:00100000:0.0:1734016143.791537:0:9611:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89fc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8216:x1818247335426304:12345-192.168.201.116@tcp:106: Request processed in 104us (276us total) trans 0 rc 0/0 00000100:00100000:0.0:1734016143.791543:0:9611:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 230 00000100:00100000:3.0:1734016143.791619:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016143.791627:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335426176 00000100:00100000:3.0:1734016143.791630:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335426176 00000100:00100000:3.0:1734016143.791649:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12717 00000100:00100000:3.0:1734016143.791653:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335426176:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.791662:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.791669:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.791681:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.791683:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4788631 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000020:00010000:3.0:1734016143.791686:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016143.791686:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4788631 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000100:00100000:3.0:1734016143.791703:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335426176:12345-192.168.201.116@tcp:106: Request processed in 49us (1438us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.791708:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12717 00010000:00010000:0.0:1734016143.805359:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.805368:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.805374:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.805378:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.805384:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.805391:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4788631, setting rss=4788631; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.805407:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.805414:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.805419:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.805423:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781fa80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.805426:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.805436:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.805439:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.805445:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781fa80/0xa50fd972cb8e730d lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.805480:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333982592:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.806256:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333982848:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.806402:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016143.806408:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333982976:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.816651:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335426432 00000100:00100000:3.0:1734016143.816659:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335426432 00000100:00100000:3.0:1734016143.816693:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12718 00000100:00100000:3.0:1734016143.816700:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335426432:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.816712:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.816721:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.816748:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.816749:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4788631 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000020:00010000:3.0:1734016143.816752:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016143.816753:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4788631 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000100:00100000:3.0:1734016143.816791:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335426432:12345-192.168.201.116@tcp:106: Request processed in 92us (240us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.816797:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12718 00000100:00100000:3.0:1734016143.821975:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335426560 00000100:00100000:3.0:1734016143.821986:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335426560 00000100:00100000:3.0:1734016143.822028:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12719 00000100:00100000:3.0:1734016143.822035:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335426560:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.822048:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.822059:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.822084:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.822086:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000020:00010000:3.0:1734016143.822099:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016143.822101:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000100:00100000:3.0:1734016143.822149:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335426560:12345-192.168.201.116@tcp:106: Request processed in 114us (1697us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.822155:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12719 00010000:00010000:3.0:1734016143.826275:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826295:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826307:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826313:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826319:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.826333:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4788631, setting rss=4788631; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016143.826341:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826347:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826354:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826360:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016143.826366:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826381:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016143.826386:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826391:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e731b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.826417:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333982848:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016143.826444:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826451:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826458:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826480:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826486:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.826494:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2473888, setting rss=2473888; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016143.826503:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826510:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826516:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826521:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016143.826523:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826530:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016143.826532:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.826537:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7322 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.826551:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333982976:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016143.826622:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8982935 i_blocks 161 00000020:00010000:0.0:1734016143.826629:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.826632:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.826637:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7e70 (ffff88012dc12bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.826639:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7eb0 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.826641:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.826643:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7e70 (ffff88012dc12638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.826646:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7eb0 (ffff88012dc12638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.826657:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016143.826667:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8982935 00000080:00200008:0.0:1734016143.826684:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016143.826689:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016143.826703:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016143.826715:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e72c7 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711763f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.826723:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e72c7 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711763f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016143.826734:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016143.826737:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e72c7 00000080:00010000:0.0:1734016143.826742:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e72c7 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016143.826745:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e72c7 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc2711763f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.826751:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365d40/0xa50fd972cb8e72c7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc2711763f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016143.826768:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016143.826777:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016143.826794:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.826797:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55c30, osc lock ffff8800a9f22be0, flags 0x400001002 00000008:00010000:0.0:1734016143.826802:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.826803:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55c70, osc lock ffff8800a9f22000, flags 0x400001002 00010000:00010000:0.0:1734016143.826820:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e7329 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.826828:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e7329 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.831368:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7330 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.831383:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7330 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.831661:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335426688 00000100:00100000:3.0:1734016143.831668:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335426688 00000100:00100000:3.0:1734016143.831703:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12720 00000100:00100000:3.0:1734016143.831709:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335426688:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.831720:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.831732:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.831755:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.831757:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4788631 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000020:00010000:3.0:1734016143.831760:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016143.831761:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4788631 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000100:00100000:3.0:1734016143.831800:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335426688:12345-192.168.201.116@tcp:106: Request processed in 91us (235us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.831806:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12720 00000100:00100000:3.0:1734016143.838387:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335426816 00000100:00100000:3.0:1734016143.838780:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335426816 00000100:00100000:3.0:1734016143.838814:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12721 00000100:00100000:3.0:1734016143.838821:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335426816:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.838832:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.838841:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.838867:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.838869:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000020:00010000:3.0:1734016143.838872:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016143.838873:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000100:00100000:3.0:1734016143.838910:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335426816:12345-192.168.201.116@tcp:106: Request processed in 90us (713us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.838916:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12721 00000080:00200000:0.0:1734016143.843233:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8982935 i_blocks 161 00000020:00010000:0.0:1734016143.843240:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.843242:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.843247:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55c30 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.843250:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55c70 (ffff88012dc12bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.843252:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.843254:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55c30 (ffff88012dc12638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.843257:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55c70 (ffff88012dc12638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.843269:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016143.843288:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016143.843293:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016143.843304:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016143.843321:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.843324:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55c30, osc lock ffff8800a9f22000, flags 0x400001002 00000008:00010000:0.0:1734016143.843329:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.843330:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55c70, osc lock ffff8800a9f22be0, flags 0x400001002 00010000:00010000:0.0:1734016143.843351:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.843358:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.843403:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.843409:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.843446:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333983360:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.843516:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016143.843522:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333983488:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.849280:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335426944 00000100:00100000:3.0:1734016143.849288:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335426944 00000100:00100000:3.0:1734016143.849322:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12722 00000100:00100000:3.0:1734016143.849328:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335426944:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.849341:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.849351:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.849375:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.849377:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4788631 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000020:00010000:3.0:1734016143.849381:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016143.849383:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4788631 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 160 00000100:00100000:3.0:1734016143.849424:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335426944:12345-192.168.201.116@tcp:106: Request processed in 97us (509us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.849430:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12722 00000100:00100000:3.0:1734016143.849492:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.849507:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335427072 00000100:00100000:3.0:1734016143.849510:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335427072 00000100:00100000:3.0:1734016143.849523:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12723 00000100:00100000:3.0:1734016143.849527:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335427072:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.849533:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.849544:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.849559:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.849560:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000020:00010000:3.0:1734016143.849563:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016143.849565:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2473888 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000100:00100000:3.0:1734016143.849579:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335427072:12345-192.168.201.116@tcp:106: Request processed in 51us (87us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.849584:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12723 00010000:00010000:0.0:1734016143.856721:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.856736:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.856749:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.856755:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.856760:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.856774:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4788631, setting rss=4788631; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.856781:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.856787:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.856793:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.856804:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c21c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.856809:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.856818:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.856823:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.856828:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e7337 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.856864:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333983360:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.858244:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.858255:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.858264:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.858269:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.858275:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.858284:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2473888, setting rss=2473888; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.859078:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.859085:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.859101:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.859106:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0000 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.859109:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.859118:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.859122:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.859129:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e733e lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.859152:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333983488:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.863927:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016143.864769:0:4749:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016143.864777:0:4749:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.864788:0:4749:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1440 already unused, calling callback (ffffffffa10571e0) 00010000:00010000:0.0:1734016143.864792:0:4749:0:(ldlm_request.c:1579:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.864799:0:4749:0:(ldlm_request.c:1589:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.864807:0:4749:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.864830:0:4749:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.864865:0:4749:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 2/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.864870:0:4749:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016143.864896:0:4749:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.864901:0:4749:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c1440/0xa50fd972cb8e72ea lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117669 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.864916:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016143.864918:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016143.864919:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016143.864933:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016143.864948:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333983872:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016143.870046:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333983872:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016143.878209:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016143.879625:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016143.879631:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.879638:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.879643:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.879664:0:5248:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.879678:0:5248:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800a8a55cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.879716:0:5248:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4784128 4788223) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016143.879769:0:5248:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a934f800 - object 0x240000400:6 offset 4784128<>4788631 00000100:00100000:0.0:1734016143.879814:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016143.879825:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333984128:192.168.201.116@tcp:4:fsx.0 00000100:00100000:0.0:1734016143.881954:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333984000:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.881980:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.881995:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.882002:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.888827:0:1981:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a55cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.889325:0:1981:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e72b9 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc2711762a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.889346:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333984128:192.168.201.116@tcp:4:fsx.0 00000100:00100000:3.0:1734016143.893792:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016143.894276:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.894285:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.894307:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.894319:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4788631, setting rss=4788631; kms=4788631, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.894334:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.894340:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.894368:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.894375:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.894941:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.894960:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333984384:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.895010:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a550c0 going to sleep for 11 seconds 00000100:00100000:3.0:1734016143.899555:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335427584 00000100:00100000:3.0:1734016143.899561:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335427584 00000100:00100000:3.0:1734016143.899600:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12727 00000100:00100000:3.0:1734016143.899606:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335427584:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016143.899619:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e72c0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117631 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.900381:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016143.900389:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335427584:12345-192.168.201.116@tcp:104: Request processed in 781us (1151us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.900395:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12727 00000100:00100000:0.0:1734016143.904077:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333984384:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.904110:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.904119:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.904126:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.906029:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335427712 00000100:00100000:3.0:1734016143.906037:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335427712 00000100:00100000:3.0:1734016143.906075:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12728 00000100:00100000:3.0:1734016143.906082:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335427712:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016143.906100:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.906109:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.906116:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.906125:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.906133:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.906141:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.906421:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c2f40) 00000100:00100000:3.0:1734016143.906484:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335427712:12345-192.168.201.116@tcp:105: Request processed in 401us (528us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.906490:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12728 00010000:00010000:0.0:1734016143.906539:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.906547:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.906567:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.906578:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 2473888, setting rss=2473888; kms=2473888, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.906587:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.906592:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.907625:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333984768:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016143.907682:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016143.907687:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333984640:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016143.912829:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333984640:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016143.914052:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333984768:192.168.201.116@tcp:10:fsx.0 00000008:00010000:3.0:1734016143.927105:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7f00 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.927168:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7f00 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.927190:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333985024:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016143.928810:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333985152:192.168.201.116@tcp:4:fsx.0 00000008:00010000:3.0:1734016143.934335:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.934377:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.934398:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333985152:192.168.201.116@tcp:4:fsx.0 00000100:00100000:0.0:1734016143.939360:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333985280:192.168.201.116@tcp:44:fsx.0 00000080:00200000:0.0:1734016143.939391:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016143.939405:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016143.948329:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 184 00000080:00200000:0.0:1734016143.948343:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016143.948444:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016143.948450:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016143.948494:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016143.948507:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016143.948545:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016143.948552:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016143.948560:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016143.948576:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333985536:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.948636:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a550c0 going to sleep for 11 seconds 00000100:00100000:0.0:1734016143.951185:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333985536:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.951207:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.951216:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.951229:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.951240:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.951250:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.951257:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.951263:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016143.951273:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a934f100 x1818247333985536/t0(0) o101->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016159 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016143.951283:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016143.951293:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.951302:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016143.951346:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016143.951350:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e7353 00000080:00010000:0.0:1734016143.951354:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016143.951357:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7353 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016143.951359:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.951370:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016143.951398:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016143.951409:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016143.951426:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.951429:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a550f0, osc lock ffff8800a9f22850, flags 0x400001002 00000008:00010000:0.0:1734016143.951434:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.951435:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55130, osc lock ffff8800a9f22e40, flags 0x400001002 00010000:00010000:0.0:1734016143.951447:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951455:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951539:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951545:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951559:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951566:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951577:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951582:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016143.951594:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 184 00000020:00010000:0.0:1734016143.951599:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8248 (ffff8800a8a54908/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.951602:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8248 (ffff8800a8a54908/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.951604:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a550f0 (ffff8800a8a54368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.951607:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55130 (ffff88012dc12cf8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.951608:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8248 (ffff8800a8a54908/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.951610:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a550f0 (ffff8800a8a54908/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.951613:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55130 (ffff8800a8a54908/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.951620:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016143.951635:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016143.951639:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016143.951642:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016143.951646:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.951647:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a550f0, osc lock ffff8800a9f22e40, flags 0x400001002 00000008:00010000:0.0:1734016143.951650:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.951651:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55130, osc lock ffff8800a9f22850, flags 0x400001002 00010000:00010000:0.0:1734016143.951656:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951662:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951669:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951674:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951683:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951688:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951694:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.951699:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016143.951706:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 184 00000020:00010000:0.0:1734016143.951709:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8248 (ffff8800a8a54908/0) at cl_lock_release():233 00000020:00010000:0.0:1734016143.951710:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8248 (ffff8800a8a54908/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.951712:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a550f0 (ffff88012dc12cf8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.951714:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55130 (ffff8800a8a54368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016143.951716:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8248 (ffff8800a8a54908/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.951718:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a550f0 (ffff8800a8a54908/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016143.951721:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55130 (ffff8800a8a54908/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016143.951725:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016143.951729:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9551825 00000080:00200008:0.0:1734016143.951740:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016143.951745:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016143.951756:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016143.951762:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016143.951789:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016143.951795:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016143.951801:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016143.951817:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333985664:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.951903:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a550c0 going to sleep for 11 seconds 00000100:00100000:0.0:1734016143.959201:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333985664:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.959222:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.959231:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.959247:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.959255:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.959264:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.959271:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016143.959276:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016143.959283:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a934f100 x1818247333985664/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016159 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016143.959292:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016143.959301:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.959311:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016143.959361:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016143.959365:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e735a 00000080:00010000:0.0:1734016143.959368:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016143.959371:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e735a from it ffff8801369fbcd8 00010000:00010000:0.0:1734016143.959373:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016143.959379:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016143.959412:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016143.959423:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016143.959446:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.959449:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a550f0, osc lock ffff8800a9f22850, flags 0x400001002 00000008:00010000:0.0:1734016143.959456:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016143.959457:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55130, osc lock ffff8800a9f22e40, flags 0x400001002 00010000:00010000:0.0:1734016143.959514:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.959522:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.959581:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.959587:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.959624:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333985920:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016143.959685:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016143.959691:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333985792:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.972554:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335427968 00000100:00100000:3.0:1734016143.972557:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335427968 00000100:00100000:3.0:1734016143.972854:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12730 00000100:00100000:3.0:1734016143.972858:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f85a050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335427968:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.972865:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.972871:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.972883:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 91, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.972885:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 184 00000020:00010000:3.0:1734016143.972888:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016143.972889:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 184 00000100:00100000:3.0:1734016143.972905:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f85a050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335427968:12345-192.168.201.116@tcp:106: Request processed in 47us (509us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.972909:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12730 00000100:00100000:0.0:1734016143.976763:0:1982:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0000-osc-ffff88012a571000: committing for last_committed 4294968090 gen 2 00010000:00010000:0.0:1734016143.976799:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976815:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976829:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976836:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976842:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.976856:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5357521, setting rss=5357521; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.976864:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976870:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976878:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976884:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3a80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.976889:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976900:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.976906:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976916:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3a80/0xa50fd972cb8e7361 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.976958:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333985792:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016143.976975:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976982:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976990:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.976995:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.977001:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016143.977008:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016143.977016:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.977021:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.977031:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.978807:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1b00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016143.978814:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.978824:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016143.978829:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016143.978836:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7368 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016143.978853:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333985920:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016143.986096:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335428096 00000100:00100000:3.0:1734016143.986105:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335428096 00000100:00100000:3.0:1734016143.986141:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12731 00000100:00100000:3.0:1734016143.986147:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335428096:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.986161:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.986171:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.986198:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 91, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.986201:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 184 00000020:00010000:3.0:1734016143.986203:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016143.986205:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 184 00000100:00100000:3.0:1734016143.986251:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335428096:12345-192.168.201.116@tcp:106: Request processed in 104us (853us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.986256:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12731 00000100:00100000:3.0:1734016143.986335:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016143.986352:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335428224 00000100:00100000:3.0:1734016143.986355:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335428224 00000100:00100000:3.0:1734016143.986371:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12732 00000100:00100000:3.0:1734016143.986375:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7982a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335428224:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016143.986381:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.986387:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016143.986400:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 91, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016143.986402:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 0 00000020:00010000:3.0:1734016143.986405:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016143.986406:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 0 00000100:00100000:3.0:1734016143.986422:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7982a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335428224:12345-192.168.201.116@tcp:106: Request processed in 46us (87us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016143.986427:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12732 00010000:00010000:3.0:1734016143.995048:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.995063:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.995075:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.995081:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.995097:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016143.995115:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5357521, setting rss=5357521; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016143.995122:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.995128:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.995135:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.995140:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016143.995146:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.995156:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016143.995161:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016143.995166:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e736f lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016143.996164:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333986048:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016144.000262:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 184 00000020:00010000:0.0:1734016144.000269:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.000272:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.000277:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7cf0 (ffff88012dc12ea8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.000280:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7d30 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.000282:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.000284:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7cf0 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.000289:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7d30 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.000302:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.000313:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9551825 00000080:00200008:0.0:1734016144.000363:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 3958771, type: SEEK_SET 00000080:00200008:0.0:1734016144.000368:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 3958771, type: SEEK_SET, rc = 3958771 00000080:00200008:0.0:1734016144.000376:0:10406:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 3958771, count: 7035 00000080:00200000:0.0:1734016144.000381:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016144.000392:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016144.000401:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 966 00000080:00200000:0.0:1734016144.000411:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 967 00000080:00200000:0.0:1734016144.000416:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 968 00000080:00200008:0.0:1734016144.000435:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016144.000442:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016144.000457:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016144.000501:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.000513:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016144.000523:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016144.000525:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e735a 00000080:00010000:0.0:1734016144.000530:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e735a from it ffff8801369fbcd8 00010000:00010000:0.0:1734016144.000532:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.000537:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016144.000555:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.000560:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.000576:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.000579:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f22130, flags 0x400001002 00000008:00010000:0.0:1734016144.000584:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.000585:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f22980, flags 0x400001002 00010000:00010000:0.0:1734016144.000603:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.000609:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.000660:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.000665:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.000689:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333986432:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016144.000758:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.000765:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333986304:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.008984:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.009030:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.009052:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335428352 00000100:00100000:3.0:1734016144.009059:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335428352 00000100:00100000:3.0:1734016144.009099:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335428480 00000100:00100000:3.0:1734016144.009101:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335428480 00000100:00100000:3.0:1734016144.009112:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12733 00000100:00100000:3.0:1734016144.009117:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335428352:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.009130:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.009139:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.009163:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 91, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.009165:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 0 00000020:00010000:3.0:1734016144.009168:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016144.009169:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 0 00000100:00100000:3.0:1734016144.009207:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335428352:12345-192.168.201.116@tcp:106: Request processed in 89us (231us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.009213:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12733 00000100:00100000:3.0:1734016144.009226:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12734 00000100:00100000:3.0:1734016144.009229:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335428480:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.009234:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.009239:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.009249:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 91, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.009250:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 184 00000020:00010000:3.0:1734016144.009253:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016144.009254:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 184 00000100:00100000:3.0:1734016144.009268:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335428480:12345-192.168.201.116@tcp:106: Request processed in 38us (238us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.009271:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12734 00010000:00010000:0.0:1734016144.012260:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012276:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012287:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012292:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012299:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.012311:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5357521, setting rss=5357521; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.012317:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012322:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012328:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012333:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.012338:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012347:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.012350:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012355:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e737d lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.012925:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333986304:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.012946:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012953:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012960:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012964:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.012970:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.012977:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.012999:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.013005:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.013010:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.013015:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c2ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.013017:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.013023:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.013026:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.013030:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7384 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.013043:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333986432:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.016743:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.016792:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.016814:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335428608 00000100:00100000:3.0:1734016144.016820:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335428608 00000100:00100000:3.0:1734016144.016847:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335428736 00000100:00100000:3.0:1734016144.016849:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335428736 00000100:00100000:3.0:1734016144.016860:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12735 00000100:00100000:3.0:1734016144.016866:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335428608:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.016879:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.016888:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.016910:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 91, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.016912:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 184 00000020:00010000:3.0:1734016144.016915:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016144.016916:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 184 00000100:00100000:3.0:1734016144.016951:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335428608:12345-192.168.201.116@tcp:106: Request processed in 86us (218us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.016956:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12735 00000100:00100000:3.0:1734016144.016966:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12736 00000100:00100000:3.0:1734016144.016970:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335428736:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.016975:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.016981:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.016991:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 91, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.016992:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 0 00000020:00010000:3.0:1734016144.016994:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016144.016995:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 0 00000100:00100000:3.0:1734016144.017006:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335428736:12345-192.168.201.116@tcp:106: Request processed in 34us (214us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.017009:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12736 00000080:00200000:3.0:1734016144.023290:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 184 00000020:00010000:3.0:1734016144.023295:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016144.023298:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.023303:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7cf0 (ffff88012dc12ea8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.023307:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7d30 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.023309:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.023311:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7cf0 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.023314:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7d30 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016144.023325:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016144.023333:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9551825 00000080:00200008:3.0:1734016144.023349:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016144.023354:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016144.023369:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016144.023381:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016144.023389:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016144.023398:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:3.0:1734016144.023401:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e7353 00000080:00010000:3.0:1734016144.023405:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7353 from it ffff8801369fbcd8 00010000:00010000:3.0:1734016144.023408:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016144.023414:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016144.023429:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016144.023439:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016144.023458:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016144.023461:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d44b0, osc lock ffff8800a8a26980, flags 0x400001002 00000008:00010000:3.0:1734016144.023486:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016144.023488:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d44f0, osc lock ffff8800a8a26ab0, flags 0x400001002 00010000:00010000:3.0:1734016144.023497:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023504:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023512:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023519:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023532:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023537:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023543:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023548:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016144.023557:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 184 00000020:00010000:3.0:1734016144.023560:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016144.023562:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.023564:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d44b0 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.023566:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d44f0 (ffff88012dc12ea8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.023568:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.023570:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d44b0 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.023572:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d44f0 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016144.023576:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016144.023585:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016144.023588:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016144.023591:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016144.023595:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016144.023596:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d44b0, osc lock ffff8800a8a26ab0, flags 0x400001002 00000008:00010000:3.0:1734016144.023598:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016144.023599:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d44f0, osc lock ffff8800a8a26980, flags 0x400001002 00010000:00010000:3.0:1734016144.023604:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023611:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023617:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023623:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023630:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023636:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023640:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023646:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016144.023653:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 184 00000020:00010000:3.0:1734016144.023655:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_release():233 00000020:00010000:3.0:1734016144.023657:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.023659:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d44b0 (ffff88012dc12ea8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.023661:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d44f0 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.023662:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.023664:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d44b0 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.023666:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d44f0 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016144.023670:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016144.023674:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9551825 00000080:00200008:3.0:1734016144.023721:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 5318616, type: SEEK_SET 00000080:00200008:3.0:1734016144.023724:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 5318616, type: SEEK_SET, rc = 5318616 00000080:00200008:3.0:1734016144.023731:0:10406:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 5318616, count: 4464 00000080:00200000:3.0:1734016144.023736:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 5318616, bytes: 4464 00000080:00200000:3.0:1734016144.023739:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:3.0:1734016144.023744:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:3.0:1734016144.023746:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [5318616, 5323080) 0 0 00000080:00200000:3.0:1734016144.023748:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1734016144.023755:0:10406:0:(file.c:1893:ll_file_io_generic()) Range lock [1298, 1299] 00020000:00200000:3.0:1734016144.023761:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [5318616, 5323080) 5323080, 4464 00020000:00200000:3.0:1734016144.023765:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016144.023777:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1298, 1299] 00000008:00010000:3.0:1734016144.023784:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016144.023786:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58ba330, osc lock ffff8800a8a26980, flags 0x0 00010000:00010000:3.0:1734016144.023800:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023806:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1122304->1130495] (req 1122304->1130495) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.023828:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1122304->1130495] (req 1122304->1130495) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.023844:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333986816:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.023888:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4240 going to sleep for 11 seconds 00000100:00100000:3.0:1734016144.029615:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016144.031121:0:4749:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016144.031128:0:4749:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.031135:0:4749:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c2f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.031140:0:4749:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e734c lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc271176d2 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.031163:0:4749:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016144.031176:0:4749:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000100:00100000:0.0:1734016144.031443:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333986944:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016144.038997:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333986816:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.039027:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1122304->1130495] (req 1122304->1130495) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.039036:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1122304->1130495] (req 1122304->1130495) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.039043:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1122304->1130495] (req 1122304->1130495) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.041596:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016144.042689:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.042699:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.042721:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.042732:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.042743:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.042749:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.042764:0:10406:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [5318616, 5323080) 00000080:00200000:0.0:1734016144.042797:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 1298 of 2008 to 2088 bytes 00000100:00100000:0.0:1734016144.042930:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:0.0:1734016144.042942:0:1981:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55180 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.042976:0:1981:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1122304 1126399) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016144.043004:0:1981:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a934f100 - object 0x280000400:8 offset 1122304<>1126400 00000100:00100000:0.0:1734016144.043020:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333987200:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016144.050720:0:1981:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a55180 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.051306:0:1981:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55180 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.051327:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333987200:192.168.201.116@tcp:3:fsx.0 00000100:00100000:0.0:1734016144.051346:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333986944:192.168.201.116@tcp:103:ldlm_bl.0 00000080:00200000:0.0:1734016144.055210:0:10406:0:(rw26.c:1006:ll_write_end()) pos 5320704, len 2376, copied 2376 00000080:00200000:0.0:1734016144.055218:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a78cd000[3 ffff8800b3a5cdc8 2 1 ffff88012dbec4f0] 00000080:00200000:0.0:1734016144.055221:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a658c0 1fffff0000080d 3:0 ffff8800a78cd000 1299 lru 00000080:00200000:0.0:1734016144.055227:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a78cd060 275: 1< 1 - - > 2< 1126400 0 4096 0x0 0x40009 | ffff8800aa960708 ffff8800a90061a0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016144.055229:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a78cd000 00000080:00200000:0.0:1734016144.055230:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:0.0:1734016144.055237:0:10406:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 2, from 2008, to 2376 00000008:00010000:0.0:1734016144.055262:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800a8a55180 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.055278:0:10406:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800a8a55180 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.055288:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134248, count 2, dirtied 2 00000080:00200000:0.0:1734016144.055292:0:10406:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 2 pages 4464 bytes, tot: 4464 00000080:00200000:0.0:1734016144.055294:0:10406:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 4464 00000020:00010000:0.0:1734016144.055306:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b63203f8 (ffff88012dc12bd8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.055308:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63203f8 (ffff88012dc12bd8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.055311:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b58ba330 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016144.055316:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.055323:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016144.055336:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63203f8 (ffff88012dc12bd8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.055337:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b58ba330 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.055343:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 4464 bytes 00000080:00200000:0.0:1734016144.055348:0:10406:0:(file.c:1930:ll_file_io_generic()) Range unlock [1298, 1299] 00000080:00200000:0.0:1734016144.055357:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016144.055360:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 4464, restart: 0 00000080:00200000:0.0:1734016144.055370:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 184 00000080:00200000:0.0:1734016144.055374:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 4464 00000080:00200008:0.0:1734016144.055392:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016144.055397:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016144.055411:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016144.055423:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.055431:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016144.055439:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016144.055442:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e735a 00000080:00010000:0.0:1734016144.055446:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e735a from it ffff8801369fbcd8 00010000:00010000:0.0:1734016144.055448:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.055454:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016144.055505:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.055513:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.055528:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.055531:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55c30, osc lock ffff8800a9f22720, flags 0x400001002 00000008:00010000:0.0:1734016144.055534:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.055535:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55c70, osc lock ffff8800a9f22390, flags 0x400001002 00010000:00010000:0.0:1734016144.055552:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.055558:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.055606:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.055613:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.055620:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.055627:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.055638:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016144.055648:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333987584:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.058008:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016144.061215:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.061230:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.061241:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.061247:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.061253:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.061265:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5357521, setting rss=5357521; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.061821:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.061828:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.061835:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.061841:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3600 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.061845:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.061855:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.061859:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.061865:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3600/0xa50fd972cb8e73a0 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.061901:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333987584:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016144.061989:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 184 00000020:00010000:0.0:1734016144.061994:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.061997:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.062002:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55c30 (ffff88012bf7f3f8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.062005:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55c70 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.062008:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbec490 (ffff88012dc12bd8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.062009:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55c30 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.062012:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55c70 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.062022:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.062037:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016144.062043:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.062050:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.062063:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.062065:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55c30, osc lock ffff8800a9f22390, flags 0x400001002 00000008:00010000:0.0:1734016144.062070:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.062071:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55c70, osc lock ffff8800a9f22720, flags 0x400001002 00010000:00010000:0.0:1734016144.062093:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.062101:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.062146:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.062153:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.062160:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.062166:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.062186:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333987712:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.066412:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016144.071564:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.071579:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.071592:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.071598:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.071604:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.071617:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5357521, setting rss=5357521; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.071638:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.071644:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.071649:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.071655:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.071660:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.071670:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.071675:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.071685:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e73a7 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.073776:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333987712:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016144.079057:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 6/1,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.079072:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016144.152728:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8801369d49c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016144.153016:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8801369d49c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.153040:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333987968:192.168.201.116@tcp:3:fsx.0 00000080:00200000:0.0:1734016144.153168:0:10406:0:(rw26.c:1006:ll_write_end()) pos 1070938, len 2214, copied 2214 00000080:00200000:0.0:1734016144.153175:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a78cd0d8[3 ffff8800b3a5ce90 2 1 ffff88012dbec4f0] 00000080:00200000:0.0:1734016144.153178:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a66ec0 1fffff0000082d 2:0 ffff8800a78cd0d8 261 lru 00000080:00200000:0.0:1734016144.153185:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a78cd138 261: 1< 1 - - > 2< 1069056 0 4096 0x0 0x40009 | ffff8800aa965cc8 ffff8800aa37b040 > 3< 0 0 > 4< 0 0 8 113500160 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016144.153186:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a78cd0d8 00000080:00200000:0.0:1734016144.153187:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:0.0:1734016144.153190:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 262 of 0 to 4096 bytes 00000080:00200000:0.0:1734016144.153197:0:10406:0:(rw26.c:1006:ll_write_end()) pos 1073152, len 4096, copied 4096 00000080:00200000:0.0:1734016144.153200:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a7493870[3 ffff8800b3a5ce90 2 1 ffff88012dbec4f0] 00000080:00200000:0.0:1734016144.153202:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002b21240 1fffff00000825 2:0 ffff8800a7493870 262 lru 00000080:00200000:0.0:1734016144.153206:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a74938d0 262: 1< 1 - - > 2< 1073152 0 4096 0x0 0x40009 | ffff8800aa965cc8 ffff8800aa37b040 > 3< 0 0 > 4< 0 0 8 113500160 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016144.153208:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a7493870 00000080:00200000:0.0:1734016144.153208:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:0.0:1734016144.153209:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 263 of 0 to 580 bytes 00000080:00200000:0.0:1734016144.153217:0:10406:0:(rw26.c:1006:ll_write_end()) pos 1077248, len 580, copied 580 00000080:00200000:0.0:1734016144.153219:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a7493948[3 ffff8800b3a5ce90 2 1 ffff88012dbec4f0] 00000080:00200000:0.0:1734016144.153221:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002b21280 1fffff00000825 2:0 ffff8800a7493948 263 lru 00000080:00200000:0.0:1734016144.153225:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a74939a8 263: 1< 1 - - > 2< 1077248 0 4096 0x0 0x40009 | ffff8800aa965cc8 ffff8800aa37b040 > 3< 0 0 > 4< 0 0 8 113500160 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016144.153226:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a7493948 00000080:00200000:0.0:1734016144.153227:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 3. 00000080:00200000:0.0:1734016144.153233:0:10406:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 3, from 1882, to 580 00000008:00010000:0.0:1734016144.153257:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800a8a559c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.153279:0:10406:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800a8a559c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.153286:0:10406:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800a8a559c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.153295:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134a48, count 3, dirtied 3 00000080:00200000:0.0:1734016144.153299:0:10406:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 3 pages 6890 bytes, tot: 6890 00000080:00200000:0.0:1734016144.153301:0:10406:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 6890 00000020:00010000:0.0:1734016144.153315:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b63203f8 (ffff88012dc12bd8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.153318:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63203f8 (ffff88012dc12bd8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.153321:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b38119b0 (ffff88012bf7f3f8/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016144.153327:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.153334:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016144.153347:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63203f8 (ffff88012dc12bd8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.153350:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b38119b0 (ffff88012dc12bd8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.153355:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 6890 bytes 00000080:00200000:0.0:1734016144.153361:0:10406:0:(file.c:1930:ll_file_io_generic()) Range unlock [261, 263] 00000080:00200000:0.0:1734016144.153369:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016144.153372:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 6890, restart: 0 00000080:00200000:0.0:1734016144.153379:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 184 00000080:00200000:0.0:1734016144.153384:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 6890 00000080:00200008:0.0:1734016144.153400:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016144.153406:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016144.153420:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016144.153432:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.153441:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016144.153452:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016144.153454:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e735a 00000080:00010000:0.0:1734016144.153459:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e735a from it ffff8801369fbcd8 00010000:00010000:0.0:1734016144.153462:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.153558:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000100:00100000:0.0:1734016144.154920:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016144.154934:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333988096:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.160312:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335429376 00000100:00100000:3.0:1734016144.160321:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335429376 00000100:00100000:3.0:1734016144.160359:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12741 00000100:00100000:3.0:1734016144.160365:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335429376:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.160379:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.160390:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.160414:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1026, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.160416:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 184 00000020:00010000:3.0:1734016144.160419:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016144.160420:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 184 00000100:00100000:3.0:1734016144.160483:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335429376:12345-192.168.201.116@tcp:106: Request processed in 116us (276us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.160490:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12741 00010000:00010000:0.0:1734016144.164992:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.165012:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.165025:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.165038:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.165044:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.165056:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5357521, setting rss=5357521; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.166138:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.166148:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.166154:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.166160:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781eac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.166166:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.166176:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.166181:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.166187:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781eac0/0xa50fd972cb8e73ae lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.166233:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333988096:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016144.167319:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.167334:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333988224:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.170358:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335429504 00000100:00100000:3.0:1734016144.170366:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335429504 00000100:00100000:3.0:1734016144.170400:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12742 00000100:00100000:3.0:1734016144.170407:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335429504:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.170421:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.170432:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.170455:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1026, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.170457:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 184 00000020:00010000:3.0:1734016144.170460:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016144.170462:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 184 00000100:00100000:3.0:1734016144.170689:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335429504:12345-192.168.201.116@tcp:106: Request processed in 281us (449us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.170697:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12742 00010000:00010000:0.0:1734016144.173648:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.173665:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.173677:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.173683:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.173689:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.173701:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5357521, setting rss=5357521; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.173722:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.173727:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.173738:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.173743:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781d8c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.173749:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.173760:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.173766:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.173771:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781d8c0/0xa50fd972cb8e73b5 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.173796:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333988224:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.178128:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335429632 00000100:00100000:3.0:1734016144.178138:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335429632 00000100:00100000:3.0:1734016144.178182:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12743 00000100:00100000:3.0:1734016144.178189:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335429632:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016144.178202:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.178269:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016144.178275:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335429632:12345-192.168.201.116@tcp:104: Request processed in 87us (258us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.178281:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12743 00010000:00010000:0.0:1734016144.178439:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016144.178445:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.178453:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.178459:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.178665:0:5248:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a559c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.178684:0:5248:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800a8a559c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.178734:0:5248:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1069056 1073151) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016144.178805:0:5248:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a934f100 - object 0x240000400:6 offset 1069056<>1081344 00000100:00100000:0.0:1734016144.178839:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016144.178851:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333988480:192.168.201.116@tcp:4:fsx.0 00000008:00010000:0.0:1734016144.187803:0:1981:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a559c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.187931:0:1981:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a559c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.187948:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333988480:192.168.201.116@tcp:4:fsx.0 00000080:00200000:3.0:1734016144.193192:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1734016144.193247:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.193256:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1734016144.193296:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.193304:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7345 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc271176cb expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.193324:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016144.193326:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016144.193329:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016144.199005:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016144.199022:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335429760 00000100:00100000:3.0:1734016144.199030:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335429760 00000100:00100000:3.0:1734016144.199072:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12744 00000100:00100000:3.0:1734016144.199079:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f858850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335429760:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016144.199100:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2244608->2248703] (req 2244608->2248703) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.199109:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [2244608->2248703] (req 2244608->2248703) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.199117:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.199125:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.199132:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.199140:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.200066:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a781fcc0) 00000100:00100000:3.0:1734016144.201559:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f858850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6465:x1818247335429760:12345-192.168.201.116@tcp:105: Request processed in 2475us (2656us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.201571:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12744 00000100:00100000:3.0:1734016144.205692:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247333988608:192.168.201.116@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1734016144.208139:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 6/1,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.208154:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.208354:0:1981:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55e40 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.208401:0:1981:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2244608 2248703) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016144.208504:0:1981:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a934e300 - object 0x240000400:6 offset 1904640<>4194304 00000100:00100000:0.0:1734016144.208523:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016144.208534:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333988864:192.168.201.116@tcp:3:fsx.0 00000080:00200000:0.0:1734016144.252298:0:10406:0:(rw26.c:1006:ll_write_end()) pos 2247212, len 1101, copied 1101 00000080:00200000:0.0:1734016144.252306:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a89bfaf8[3 ffff8800b3a5cdc8 2 1 ffff88012dbec2a8] 00000080:00200000:0.0:1734016144.252308:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a5b980 1fffff0000082d 2:0 ffff8800a89bfaf8 548 lru 00000080:00200000:0.0:1734016144.252316:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a89bfb58 548: 1< 1 - - > 2< 2244608 0 4096 0x0 0x40009 | ffff88012a41dcc8 ffff8800a9006000 > 3< 0 0 > 4< 0 0 8 489881600 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016144.252318:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a89bfaf8 00000080:00200000:0.0:1734016144.252319:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:0.0:1734016144.252325:0:10406:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 1, from 2604, to 3705 00000008:00010000:0.0:1734016144.252366:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800a8a55300 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.252383:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134248, count 1, dirtied 1 00000080:00200000:0.0:1734016144.252387:0:10406:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 1 pages 1101 bytes, tot: 1101 00000080:00200000:0.0:1734016144.252389:0:10406:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 1101 00000020:00010000:0.0:1734016144.252407:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b6320a88 (ffff88012dc12e18/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.252410:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6320a88 (ffff88012dc12e18/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.252414:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801376bac30 (ffff88012dc121b8/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016144.252422:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252431:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016144.252449:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6320a88 (ffff88012dc12e18/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.252453:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801376bac30 (ffff88012dc12e18/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.252460:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 1101 bytes 00000080:00200000:0.0:1734016144.252491:0:10406:0:(file.c:1930:ll_file_io_generic()) Range unlock [548, 548] 00000080:00200000:0.0:1734016144.252500:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016144.252503:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 1101, restart: 0 00000080:00200000:0.0:1734016144.252516:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 208 00000080:00200000:0.0:1734016144.252521:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 1101 00000080:00200008:0.0:1734016144.252543:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016144.252548:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016144.252564:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016144.252578:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.252587:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016144.252597:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016144.252600:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e735a 00000080:00010000:0.0:1734016144.252605:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e735a from it ffff8801369fbcd8 00010000:00010000:0.0:1734016144.252607:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.252613:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e735a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176ee expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016144.252627:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.252641:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.252661:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.252666:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55ab0, osc lock ffff8800a9f22260, flags 0x400001002 00000008:00010000:0.0:1734016144.252675:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.252676:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55af0, osc lock ffff8800a9f225f0, flags 0x400001002 00010000:00010000:0.0:1734016144.252689:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252698:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252707:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252712:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252728:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252734:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252740:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252745:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.252762:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 208 00000020:00010000:0.0:1734016144.252764:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbec248 (ffff88012dc12e18/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.252766:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbec248 (ffff88012dc12e18/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.252768:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55ab0 (ffff88012dc121b8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.252770:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55af0 (ffff8800a8a54e18/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.252772:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbec248 (ffff88012dc12e18/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.252774:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55ab0 (ffff88012dc12e18/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.252777:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55af0 (ffff88012dc12e18/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.252785:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.252795:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016144.252799:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.252801:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.252808:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.252810:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55ab0, osc lock ffff8800a9f225f0, flags 0x400001002 00000008:00010000:0.0:1734016144.252812:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.252813:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55af0, osc lock ffff8800a9f22260, flags 0x400001002 00010000:00010000:0.0:1734016144.252817:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252824:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252830:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252836:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252844:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252851:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252856:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.252862:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.252869:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 208 00000020:00010000:0.0:1734016144.252871:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbec248 (ffff88012dc12e18/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.252873:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbec248 (ffff88012dc12e18/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.252875:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55ab0 (ffff8800a8a54e18/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.252877:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55af0 (ffff88012dc121b8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.252879:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbec248 (ffff88012dc12e18/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.252880:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55ab0 (ffff88012dc12e18/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.252883:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55af0 (ffff88012dc12e18/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.252887:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.252892:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9551825 00000080:00200008:0.0:1734016144.252902:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016144.252908:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016144.252919:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016144.252932:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.252942:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016144.252948:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016144.252949:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e7353 00000080:00010000:0.0:1734016144.252952:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7353 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016144.252955:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.252961:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016144.252972:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.252977:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.252983:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.252984:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55ab0, osc lock ffff8800a9f22260, flags 0x400001002 00000008:00010000:0.0:1734016144.252988:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.252989:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55af0, osc lock ffff8800a9f225f0, flags 0x400001002 00010000:00010000:0.0:1734016144.253005:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e73c3 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.253012:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e73c3 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.253076:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.253096:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.253138:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333989120:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.260044:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335429888 00000100:00100000:3.0:1734016144.260052:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335429888 00000100:00100000:3.0:1734016144.260099:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12745 00000100:00100000:3.0:1734016144.260106:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335429888:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.260119:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.260129:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.260154:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 562, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.260156:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 1 00000020:00010000:3.0:1734016144.260159:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016144.260160:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 1 00000100:00100000:3.0:1734016144.260201:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335429888:12345-192.168.201.116@tcp:106: Request processed in 96us (1377us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.260207:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12745 00000100:00100000:3.0:1734016144.263984:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335430016 00000100:00100000:3.0:1734016144.263993:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335430016 00000100:00100000:3.0:1734016144.264042:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12746 00000100:00100000:3.0:1734016144.264050:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335430016:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.264063:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.264076:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.264109:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 562, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.264112:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 208 00000020:00010000:3.0:1734016144.264115:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016144.264116:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 208 00000100:00100000:3.0:1734016144.264165:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335430016:12345-192.168.201.116@tcp:106: Request processed in 116us (595us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.264172:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12746 00010000:00010000:0.0:1734016144.266871:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.266889:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.266903:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.266908:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.266915:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.266929:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.266936:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.266941:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.266952:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.266958:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0480 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.266964:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.266973:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.266978:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.266984:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e73ca lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.267024:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333989120:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016144.269034:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 209 00000020:00010000:0.0:1734016144.269040:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbec248 (ffff88012dc12e18/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.269043:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbec248 (ffff88012dc12e18/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.269047:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55ab0 (ffff88012dc121b8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.269050:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55af0 (ffff8800a8a54e18/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.269052:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbec248 (ffff88012dc12e18/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.269054:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55ab0 (ffff88012dc12e18/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.269058:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55af0 (ffff88012dc12e18/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.269068:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016144.269204:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016144.269209:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016144.269222:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016144.269232:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2233, 2233] 00000008:00010000:0.0:1734016144.269244:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016144.269247:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999bb0, osc lock ffff8800a9f225f0, flags 0x0 00010000:00010000:0.0:1734016144.269264:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.269270:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4952064->4956159] (req 4952064->4956159) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.269291:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4952064->4956159] (req 4952064->4956159) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.269307:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333989248:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016144.269409:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55000 going to sleep for 11 seconds 00000100:00100000:3.0:1734016144.272360:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016144.273575:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.273589:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333989376:192.168.201.116@tcp:4:fsx.0 00000100:00100000:0.0:1734016144.280186:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333989248:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.280221:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4952064->4956159] (req 4952064->4956159) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.280231:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4952064->4956159] (req 4952064->4956159) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.280237:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4952064->4956159] (req 4952064->4956159) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.283774:0:1982:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a55300 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.284360:0:1982:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55300 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.284382:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333989376:192.168.201.116@tcp:4:fsx.0 00000080:00200000:0.0:1734016144.291208:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016144.291266:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 2/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.291275:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016144.291293:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.291300:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e73bc lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2244608->2248703) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117750 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.291316:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.291318:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.291321:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016144.291340:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333989504:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016144.298803:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016144.301322:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.301335:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.301365:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.301377:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 5357521, setting rss=5357521; kms=5357521, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.301386:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.301393:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.301414:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 217 00000100:00100000:0.0:1734016144.304304:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333989504:192.168.201.116@tcp:103:ldlm_bl.0 00000020:00010000:0.0:1734016144.313511:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b6320a88 (ffff88012dc12e18/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.313515:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6320a88 (ffff88012dc12e18/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.313520:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999bb0 (ffff8800a8a54e18/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016144.313528:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.313539:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016144.313550:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6320a88 (ffff88012dc12e18/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.313552:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999bb0 (ffff88012dc12e18/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.313566:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.313612:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016144.313618:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016144.313633:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016144.313645:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.313655:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016144.313663:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016144.313666:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e7353 00000080:00010000:0.0:1734016144.313670:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7353 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016144.313673:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.313679:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016144.313692:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.313701:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.313714:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.313717:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55030, osc lock ffff8800a9f225f0, flags 0x400001002 00000008:00010000:0.0:1734016144.313722:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.313723:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55070, osc lock ffff8800a9f22260, flags 0x400001002 00010000:00010000:0.0:1734016144.313731:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.313738:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.313748:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.313753:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.313769:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.313776:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.313830:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333989760:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.317851:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016144.326386:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.326399:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.326410:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.326415:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.326424:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.326435:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.326457:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.326569:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.326578:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.326584:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.326589:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.326601:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.326606:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.326610:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e73d8 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.326640:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333989760:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.334663:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016144.346340:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9551825 i_blocks 225 00000020:00010000:0.0:1734016144.346346:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.346349:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.346354:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7f30 (ffff8800a8a54e18/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.346357:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7f70 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.346359:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.346361:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7f30 (ffff88012dc12758/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.346365:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7f70 (ffff88012dc12758/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.346376:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.346386:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9551825 00000080:00a00000:0.0:1734016144.346443:0:10406:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800b6e4e1b0 start=0x7f17cf5cb000 end=0x7f17cf5cc000 vm_flags=0xfb 00000080:00200000:0.0:1734016144.346454:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.346480:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.346499:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.346501:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55330, osc lock ffff8800a9f22260, flags 0x400001002 00000008:00010000:0.0:1734016144.346508:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.346509:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55370, osc lock ffff8800a9f225f0, flags 0x400001002 00010000:00010000:0.0:1734016144.346529:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.346537:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.346601:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.346611:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.346624:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333990016:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.346677:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.346686:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.346691:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.350764:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.350796:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016144.350811:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335430656 00000100:00100000:3.0:1734016144.350818:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335430656 00000100:00100000:3.0:1734016144.350851:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12751 00000100:00100000:3.0:1734016144.350857:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335430656:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.350870:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.350879:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e73d1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4952064->4956159) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117765 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.350906:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.350908:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 224 00000020:00010000:3.0:1734016144.350911:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016144.350912:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5357521 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 224 00000100:00100000:3.0:1734016144.350948:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335430656:12345-192.168.201.116@tcp:106: Request processed in 91us (193us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.350954:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12751 00010000:00010000:0.0:1734016144.354718:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.354735:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.354748:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.354754:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.354760:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.354772:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5357521, setting rss=5357521; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.355098:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.355109:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.355115:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.355120:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.355124:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.355134:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.355138:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.355143:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1680/0xa50fd972cb8e73e6 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.355181:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333990016:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.361680:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.366012:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000008:00010000:3.0:1734016144.384488:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.384516:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (872448 876543) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016144.384545:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982d80 - object 0x240000400:6 offset 872448<>876544 00000100:00100000:3.0:1734016144.384560:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016144.384571:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333990528:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016144.384765:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:3.0:1734016144.384772:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8800b6ee7780 00010000:00010000:3.0:1734016144.384803:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016144.384913:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7980380 - object 0x280000400:8 offset 0<>4194304 00000100:00100000:3.0:1734016144.384927:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333990784:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016144.391816:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016144.391860:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.391887:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333990528:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016144.398632:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7f00 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016144.398671:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7f00 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.398690:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333990912:192.168.201.116@tcp:4:fsx.0 00000100:00100000:0.0:1734016144.404374:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333991040:192.168.201.116@tcp:44:fsx.0 00000080:00200000:0.0:1734016144.404410:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016144.404425:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1734016144.404532:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.404548:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333991168:192.168.201.116@tcp:16:fsx.0 00000100:00100000:0.0:1734016144.427608:0:1982:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0000-osc-ffff88012a571000: committing for last_committed 4294968096 gen 2 00000100:00100000:0.0:1734016144.427653:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333991168:192.168.201.116@tcp:16:fsx.0 00000008:00010000:3.0:1734016144.477053:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016144.477168:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.477193:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333991424:192.168.201.116@tcp:3:fsx.0 00000020:00010000:0.0:1734016144.478122:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b6320d58 (ffff88012dc12758/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.478127:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6320d58 (ffff88012dc12758/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.478130:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811130 (ffff8800a8a54e18/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016144.478138:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.478148:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016144.478162:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6320d58 (ffff88012dc12758/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.478165:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811130 (ffff88012dc12758/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.478179:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016144.478214:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 271 00000080:00200000:0.0:1734016144.478220:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016144.478230:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016144.478241:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [271, 271] 00000008:00010000:0.0:1734016144.478253:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016144.478256:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999bb0, osc lock ffff8800a9f22720, flags 0x0 00010000:00010000:0.0:1734016144.478271:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.478281:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1110016 1114111) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.478290:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 6/1,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.478300:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 6/1,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.478306:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.478382:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:0.0:1734016144.478393:0:1981:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55d80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.478412:0:1981:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1110016 1114111) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016144.478439:0:1981:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a934f800 - object 0x240000400:6 offset 1110016<>1114112 00000100:00100000:0.0:1734016144.478453:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333991680:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016144.487571:0:1981:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a55d80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.487608:0:1981:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55d80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.487628:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333991680:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016144.493403:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### no dlm lock: extent: ffff8800b6ee7780 00000008:00010000:3.0:1734016144.493671:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### no dlm lock: extent: ffff8800b6ee7780 00000100:00100000:3.0:1734016144.493691:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247333990784:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016144.498326:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016144.508739:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016144.508750:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.508764:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0240 already unused, calling callback (ffffffffa10571e0) 00010000:00010000:0.0:1734016144.508769:0:5248:0:(ldlm_request.c:1579:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.508776:0:5248:0:(ldlm_request.c:1589:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.508784:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.508805:0:5248:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781c900/0xa50fd972cb8e5f33 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115fb0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.508840:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 2/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.508844:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016144.508873:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.508878:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7353 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc271176e0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.508908:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.508909:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.508911:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016144.508919:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016144.508933:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333992064:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016144.515139:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0e00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333991936:192.168.201.116@tcp:36:fsx.0 00000080:00200000:0.0:1734016144.515176:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016144.515179:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016144.515201:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016144.515246:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1734016144.515260:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016144.515264:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55e70, osc lock ffff8800a9f22390, flags 0x0 00000008:00010000:0.0:1734016144.515267:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016144.515268:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55eb0, osc lock ffff8800a9f22980, flags 0x0 00010000:00010000:0.0:1734016144.515285:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.515297:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.515306:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.515312:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.515318:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.515327:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.515332:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.515338:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.515343:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.515347:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.515424:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333992192:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016144.515501:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247333992064:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016144.515514:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016144.515519:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333992320:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016144.519948:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333992320:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016144.519976:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247333992192:192.168.201.116@tcp:10:fsx.0 00000100:00100000:3.0:1734016144.533073:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.535235:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.535910:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12756 00000100:00100000:3.0:1734016144.535918:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335431168:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.535933:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.535942:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.535969:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 688, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.535970:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2800218 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 1 00000020:00010000:3.0:1734016144.535973:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016144.535974:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2800218 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 1 00000100:00100000:3.0:1734016144.536019:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335431168:12345-192.168.201.116@tcp:106: Request processed in 103us (795us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.536025:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12756 00000080:00200000:3.0:1734016144.552236:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6994522 i_blocks 201 00000020:00010000:3.0:1734016144.552242:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_release():233 00000020:00010000:3.0:1734016144.552245:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.552251:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7e70 (ffff8800a8a54e18/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.552254:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7eb0 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.552256:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.552257:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7e70 (ffff88012dc12758/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.552260:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7eb0 (ffff88012dc12758/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016144.552272:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016144.552291:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016144.552297:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016144.552306:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016144.552322:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016144.552324:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41b0, osc lock ffff8800a8a26ab0, flags 0x400001002 00000008:00010000:3.0:1734016144.552329:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016144.552330:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41f0, osc lock ffff8800a8a26be0, flags 0x400001002 00010000:00010000:3.0:1734016144.552351:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34b3c0/0xa50fd972cb8e7409 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.552358:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34b3c0/0xa50fd972cb8e7409 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.552408:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e7410 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.552414:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa348900/0xa50fd972cb8e7410 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.557728:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.558967:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:3.0:1734016144.577762:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6994522 i_blocks 201 00000020:00010000:3.0:1734016144.577768:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_release():233 00000020:00010000:3.0:1734016144.577771:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.577775:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41b0 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.577778:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41f0 (ffff8800a8a54e18/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.577780:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.577782:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41b0 (ffff88012dc12758/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.577786:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41f0 (ffff88012dc12758/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016144.577798:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016144.577809:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 6994522 00000080:00200008:3.0:1734016144.577825:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1734016144.577832:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:3.0:1734016144.577848:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:3.0:1734016144.577864:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016144.577874:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:3.0:1734016144.577884:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:3.0:1734016144.577887:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e73f4 00000080:00010000:3.0:1734016144.577891:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e73f4 from it ffff8801369fbcd8 00010000:00010000:3.0:1734016144.577893:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016144.577899:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1734016144.577917:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016144.577928:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016144.577945:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016144.577948:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41b0, osc lock ffff8800a8a26be0, flags 0x400001002 00000008:00010000:3.0:1734016144.577953:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016144.577954:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41f0, osc lock ffff8800a8a26ab0, flags 0x400001002 00010000:00010000:3.0:1734016144.577972:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34bcc0/0xa50fd972cb8e7417 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.577979:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34bcc0/0xa50fd972cb8e7417 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.578020:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a1c0/0xa50fd972cb8e741e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.578026:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa34a1c0/0xa50fd972cb8e741e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.588064:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.588151:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.588180:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335431680 00000100:00100000:3.0:1734016144.588188:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335431680 00000100:00100000:3.0:1734016144.588218:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335431808 00000100:00100000:3.0:1734016144.588219:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335431808 00000100:00100000:3.0:1734016144.588231:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12759 00000100:00100000:3.0:1734016144.588237:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335431680:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.588252:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.588261:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.588287:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 688, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.588289:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2800218 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 1 00000020:00010000:3.0:1734016144.588291:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016144.588293:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2800218 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 1 00000100:00100000:3.0:1734016144.588334:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335431680:12345-192.168.201.116@tcp:106: Request processed in 97us (279us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.588342:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12759 00000100:00100000:3.0:1734016144.588355:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12760 00000100:00100000:3.0:1734016144.588358:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7982680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335431808:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.588363:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.588369:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.588379:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 688, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.588380:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 200 00000020:00010000:3.0:1734016144.588382:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016144.588383:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 200 00000100:00100000:3.0:1734016144.588396:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7982680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335431808:12345-192.168.201.116@tcp:106: Request processed in 37us (245us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.588399:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12760 00000080:00200000:3.0:1734016144.608021:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6994522 i_blocks 201 00000020:00010000:3.0:1734016144.608027:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_release():233 00000020:00010000:3.0:1734016144.608030:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.608036:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41b0 (ffff8800a8a54e18/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.608040:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41f0 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.608042:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.608044:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41b0 (ffff88012dc12758/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.608047:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41f0 (ffff88012dc12758/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016144.608060:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016144.608087:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:3.0:1734016144.608092:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1734016144.608100:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:3.0:1734016144.608116:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016144.608119:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41b0, osc lock ffff8800a8a26ab0, flags 0x400001002 00000008:00010000:3.0:1734016144.608122:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1734016144.608124:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d41f0, osc lock ffff8800a8a26be0, flags 0x400001002 00010000:00010000:3.0:1734016144.608150:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e7425 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.608158:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa34af40/0xa50fd972cb8e7425 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.608211:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e742c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.608217:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa3486c0/0xa50fd972cb8e742c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.615054:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.615119:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.615144:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335431936 00000100:00100000:3.0:1734016144.615151:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335431936 00000100:00100000:3.0:1734016144.615183:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335432064 00000100:00100000:3.0:1734016144.615184:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335432064 00000100:00100000:3.0:1734016144.615196:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12761 00000100:00100000:3.0:1734016144.615202:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335431936:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.615216:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.615225:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.615253:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 688, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.615255:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2800218 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 1 00000020:00010000:3.0:1734016144.615257:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016144.615258:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2800218 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 1 00000100:00100000:3.0:1734016144.615305:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335431936:12345-192.168.201.116@tcp:106: Request processed in 103us (262us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.615310:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12761 00000100:00100000:3.0:1734016144.615323:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12762 00000100:00100000:3.0:1734016144.615327:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7982d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335432064:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.615334:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.615339:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.615349:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 688, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.615350:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 200 00000020:00010000:3.0:1734016144.615353:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016144.615354:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 200 00000100:00100000:3.0:1734016144.615369:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7982d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335432064:12345-192.168.201.116@tcp:106: Request processed in 41us (250us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.615373:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12762 00000080:00200000:3.0:1734016144.621940:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6994522 i_blocks 201 00000020:00010000:3.0:1734016144.621947:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_release():233 00000020:00010000:3.0:1734016144.621949:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.621955:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41b0 (ffff88012dc12ab8/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.621957:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369d41f0 (ffff8800a8a54e18/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016144.621959:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbeefe8 (ffff88012dc12758/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.621961:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41b0 (ffff88012dc12758/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016144.621965:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369d41f0 (ffff88012dc12758/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016144.621974:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016144.621985:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 6994522 00000080:00200008:3.0:1734016144.622045:0:10406:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8) current size 6994522, valid attrs 2068, mode 8801, uid -1227447776, gid -30720, new size 2805133, atime 1734016144.000000000, mtime 1734016144.000000000, ctime 1734016144.000000000 00000020:00100000:3.0:1734016144.622102:0:10406:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff88012a571000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1734016144.622109:0:10406:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a7980e00 x1818247333993728/t0(0) o36->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016144.622127:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333993728:192.168.201.116@tcp:36:fsx.0 00000100:00100000:3.0:1734016144.622181:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4180 going to sleep for 11 seconds 00000100:00100000:3.0:1734016144.628091:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016144.628437:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016144.628444:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.628451:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa10571e0) 00010000:00010000:0.0:1734016144.628454:0:4750:0:(ldlm_request.c:1579:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.628493:0:4750:0:(ldlm_request.c:1589:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.628501:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.628523:0:4750:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781c900/0xa50fd972cb8e5f33 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115fb0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.628554:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 2/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.628559:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016144.628584:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.628590:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e73f4 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117796 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.628613:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.628614:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.628616:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016144.639843:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333993728:192.168.201.116@tcp:36:fsx.0 00000080:00200000:0.0:1734016144.639904:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016144.639909:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016144.639928:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016144.639961:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1734016144.639978:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016144.639981:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f30, osc lock ffff8800a9f22130, flags 0x0 00000008:00010000:0.0:1734016144.639986:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016144.639987:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f70, osc lock ffff8800a9f22e40, flags 0x0 00010000:00010000:0.0:1734016144.640005:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.640017:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.640026:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.640034:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.640039:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.640049:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.640054:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.640059:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.640064:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.640074:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.641231:0:10406:0:(osc_cache.c:2696:osc_cache_truncate_start()) ### extent: ffff8800a8a55180 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.641266:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016144.641316:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:0.0:1734016144.641324:0:10406:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55180 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.682808:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016144.692976:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.692992:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.693004:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.693009:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.693015:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016144.693028:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016144.693034:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.693039:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.693046:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.693051:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016144.693056:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.693065:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016144.693069:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.693086:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7441 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.693114:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333994496:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016144.695768:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.695784:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.695795:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.695800:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.695805:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016144.695817:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2805133, setting rss=2805133; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016144.695868:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.695874:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.695880:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.695884:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016144.695889:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.695898:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016144.695903:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.695908:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e743a lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.695927:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247333994368:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016144.695988:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 2805133 i_blocks 184 00000020:00010000:0.0:1734016144.695994:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.695996:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.696001:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee79f0 (ffff8800a8a54518/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.696004:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7a30 (ffff8800a8a543f8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.696005:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.696007:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee79f0 (ffff8800a8a54128/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.696010:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7a30 (ffff8800a8a54128/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.696019:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.696036:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016144.696040:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.696047:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.696059:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.696061:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551b0, osc lock ffff8800a9f22850, flags 0x400001002 00000008:00010000:0.0:1734016144.696066:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.696068:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551f0, osc lock ffff8800a9f22be0, flags 0x400001002 00010000:00010000:0.0:1734016144.696096:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7448 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.696103:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7448 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.696158:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e744f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.696164:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e744f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.714370:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 2805133 i_blocks 184 00000020:00010000:0.0:1734016144.714377:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.714380:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.714384:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a551b0 (ffff8800a8a543f8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.714388:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a551f0 (ffff8800a8a54518/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.714389:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.714392:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a551b0 (ffff8800a8a54128/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.714408:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a551f0 (ffff8800a8a54128/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.714422:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.714433:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 2805133 00000080:00200008:0.0:1734016144.715410:0:10406:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8) current size 2805133, valid attrs 2068, mode 8801, uid -1227447776, gid -30720, new size 6922928, atime 1734016144.000000000, mtime 1734016144.000000000, ctime 1734016144.000000000 00010000:00010000:0.0:1734016144.715436:0:10406:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7433 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc271177e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.715453:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781c900/0xa50fd972cb8e5f33 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115fb0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.715503:0:10406:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7433 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc271177e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.715509:0:10406:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016144.715511:0:10406:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7433 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc271177e3 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000020:00100000:0.0:1734016144.715542:0:10406:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff88012a5ff800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1734016144.715550:0:10406:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a934c700 x1818247333995008/t0(0) o36->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016144.715569:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333995008:192.168.201.116@tcp:36:fsx.0 00000100:00100000:0.0:1734016144.715626:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55180 going to sleep for 11 seconds 00000100:00100000:0.0:1734016144.720613:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333995008:192.168.201.116@tcp:36:fsx.0 00000080:00200000:0.0:1734016144.720651:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016144.720657:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016144.720676:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016144.720694:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1734016144.720710:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016144.720715:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551b0, osc lock ffff8800a9f22be0, flags 0x0 00000008:00010000:0.0:1734016144.720720:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016144.720722:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551f0, osc lock ffff8800a9f22850, flags 0x0 00010000:00010000:0.0:1734016144.720751:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.720761:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.720787:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.720807:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333995136:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016144.721906:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a550c0 going to sleep for 11 seconds 00000100:00100000:3.0:1734016144.727512:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335432832 00000100:00100000:3.0:1734016144.727521:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335432832 00000100:00100000:3.0:1734016144.727558:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12768 00000100:00100000:3.0:1734016144.727565:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335432832:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016144.727579:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.727640:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016144.727646:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335432832:12345-192.168.201.116@tcp:104: Request processed in 82us (284us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.727652:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12768 00010000:00010000:0.0:1734016144.727762:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016144.727769:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.727778:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.727783:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.727831:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016144.728174:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016144.728216:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 2/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.728224:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016144.730268:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.730280:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e73ed lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 872448->876543) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117781 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.730307:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.730309:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.730312:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016144.730573:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333995264:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016144.735428:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333995136:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.735460:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.735874:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.735882:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.736230:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335432960 00000100:00100000:3.0:1734016144.736236:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335432960 00000100:00100000:3.0:1734016144.736311:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12769 00000100:00100000:3.0:1734016144.736318:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335432960:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016144.736328:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.736339:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.736346:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.736353:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.736360:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.736368:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.737388:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c3840) 00000100:00100000:3.0:1734016144.737517:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335432960:12345-192.168.201.116@tcp:105: Request processed in 1197us (1454us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.737526:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12769 00010000:00010000:0.0:1734016144.737914:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.737924:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.737951:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.737964:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 2805133, setting rss=2805133; kms=2805133, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.737975:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.737982:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.738015:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.738023:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.738060:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.738090:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333995392:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016144.738136:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a556c0 going to sleep for 11 seconds 00000100:00100000:0.0:1734016144.743297:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333995264:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016144.743614:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335433088 00000100:00100000:3.0:1734016144.743621:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335433088 00000100:00100000:3.0:1734016144.743653:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12770 00000100:00100000:3.0:1734016144.743659:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335433088:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016144.743673:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.743734:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016144.743739:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335433088:12345-192.168.201.116@tcp:104: Request processed in 81us (371us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.743745:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12770 00010000:00010000:0.0:1734016144.744110:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016144.744116:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.744122:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa349d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.744127:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.744149:0:4750:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016144.744163:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016144.744202:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.744213:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016144.745258:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.745266:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa349d40/0xa50fd972cb8e7399 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1122304->1130495) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc2711772d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.745303:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.745305:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.745307:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016144.745313:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.745328:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333995520:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016144.752815:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333995392:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.752844:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.752853:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.752859:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.753382:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.753393:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.753409:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.753419:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.753428:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.753433:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.755706:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333995648:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016144.761685:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333995520:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016144.762190:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333995648:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016144.779423:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333996032:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.789749:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.789759:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.789767:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.789772:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.789778:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.789785:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.789794:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.789800:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.789806:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.789811:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781fa80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.789813:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.789819:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.789822:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.789831:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fa80/0xa50fd972cb8e746b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.789862:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333996032:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016144.791124:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333996288:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.798754:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.798764:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.798771:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.798776:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.798780:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.798788:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.799330:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.799341:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.799346:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.799350:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781fcc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.799353:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.799363:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.799367:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.799372:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781fcc0/0xa50fd972cb8e7479 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.799768:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333996288:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016144.808149:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6922928 i_blocks 184 00000020:00010000:0.0:1734016144.808155:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.808158:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.808163:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7b193f0 (ffff8800a8a543f8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.808166:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7b19430 (ffff8800a8a54518/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.808168:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.808170:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7b193f0 (ffff8800a8a54128/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.808174:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7b19430 (ffff8800a8a54128/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.808185:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016144.808233:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 172 00000080:00200000:0.0:1734016144.808240:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016144.808252:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016144.808264:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [172, 172] 00000008:00010000:0.0:1734016144.808276:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016144.808278:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369996b0, osc lock ffff8800a9f22000, flags 0x0 00010000:00010000:0.0:1734016144.808295:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.808302:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [704512->708607] (req 704512->708607) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.808331:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [704512->708607] (req 704512->708607) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.808350:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333996800:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016144.808412:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55f00 going to sleep for 11 seconds 00000100:00100000:3.0:1734016144.814397:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016144.814413:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335434112 00000100:00100000:3.0:1734016144.814420:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335434112 00000100:00100000:3.0:1734016144.814452:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12778 00000100:00100000:3.0:1734016144.814458:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a89aa050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335434112:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016144.814497:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7456 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711780d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.814881:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016144.814888:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a89aa050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335434112:12345-192.168.201.116@tcp:104: Request processed in 430us (599us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.814895:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12778 00000100:00100000:3.0:1734016144.820593:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335434240 00000100:00100000:3.0:1734016144.820602:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335434240 00000100:00100000:3.0:1734016144.821280:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12779 00000100:00100000:3.0:1734016144.821288:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335434240:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016144.821302:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [704512->708607] (req 704512->708607) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.821311:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [704512->708607] (req 704512->708607) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.821322:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.821332:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.821339:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.821346:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.821363:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c2d00) 00000100:00100000:3.0:1734016144.821515:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6465:x1818247335434240:12345-192.168.201.116@tcp:105: Request processed in 227us (1027us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.821520:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12779 00000100:00100000:0.0:1734016144.823287:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247333996800:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.823308:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.823328:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.823341:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.823357:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.823362:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.823811:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:0.0:1734016144.823829:0:1982:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55f00 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.823860:0:1982:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (704512 708607) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016144.823887:0:1982:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a934ca80 - object 0x240000400:6 offset 704512<>708608 00000100:00100000:0.0:1734016144.823903:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333997184:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016144.836956:0:1982:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a55f00 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.836985:0:1982:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55f00 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.837006:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333997184:192.168.201.116@tcp:3:fsx.0 00000100:00100000:0.0:1734016144.841779:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333997312:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.841798:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016144.841807:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016144.841818:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016144.841825:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016144.841833:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016144.841839:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016144.841843:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016144.841851:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78a3100 x1818247333997312/t0(0) o101->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016160 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016144.841858:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016144.841867:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.841875:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016144.841924:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016144.841928:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e749c 00000080:00010000:0.0:1734016144.841932:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016144.841934:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e749c from it ffff8801369fbcd8 00010000:00010000:0.0:1734016144.841936:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.841942:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016144.841967:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.841977:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.841996:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.841999:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f30, osc lock ffff8800a9f23430, flags 0x400001002 00000008:00010000:0.0:1734016144.842003:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.842004:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f70, osc lock ffff8800a9f22d10, flags 0x400001002 00010000:00010000:0.0:1734016144.842025:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.842031:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.847404:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.847427:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.847437:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.847448:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.847489:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.847508:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333997440:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.859419:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.859442:0:1982:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.859460:0:1982:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.859488:0:1982:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.859502:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.859531:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.859538:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.859549:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.859555:0:1982:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.859573:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333997440:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.864270:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.864321:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.864554:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335434496 00000100:00100000:3.0:1734016144.864557:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335434496 00000100:00100000:3.0:1734016144.864573:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12780 00000100:00100000:3.0:1734016144.864579:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7982a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335434368:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016144.864592:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.864642:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016144.864648:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7982a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335434368:12345-192.168.201.116@tcp:104: Request processed in 69us (387us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.864653:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12780 00000100:00100000:3.0:1734016144.864664:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12781 00000100:00100000:3.0:1734016144.864668:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335434496:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016144.864674:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7495 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 704512->708607) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117853 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.865022:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016144.865028:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335434496:12345-192.168.201.116@tcp:104: Request processed in 359us (708us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.865034:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12781 00010000:00010000:0.0:1734016144.865063:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1734016144.865070:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.865084:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.865089:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.865136:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016144.865150:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016144.865184:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.865189:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016144.865516:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.865522:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e74a3 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117868 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.865538:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.865539:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016144.865541:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016144.865556:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333997696:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016144.865599:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.865604:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333997824:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016144.874453:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333997696:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016144.875369:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335434624 00000100:00100000:3.0:1734016144.875377:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335434624 00000100:00100000:3.0:1734016144.875423:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12782 00000100:00100000:3.0:1734016144.875430:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335434624:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016144.875444:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [6123520->6127615] (req 6123520->6127615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.875454:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [6123520->6127615] (req 6123520->6127615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.875462:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.875493:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.875505:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.875513:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.875527:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa364b40) 00000100:00100000:3.0:1734016144.875575:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335434624:12345-192.168.201.116@tcp:105: Request processed in 145us (1249us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.875582:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12782 00000100:00100000:0.0:1734016144.879435:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247333997568:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.879459:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.879517:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.879530:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.879541:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.879547:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016144.879571:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6922928 i_blocks 184 00000020:00010000:0.0:1734016144.888682:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a328a8 (ffff8800a8a54a28/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.888686:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a328a8 (ffff8800a8a54a28/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.888691:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b38117b0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016144.888702:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.888715:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016144.888730:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a328a8 (ffff8800a8a54a28/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.888734:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b38117b0 (ffff8800a8a54a28/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.888750:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.888805:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016144.888812:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016144.888829:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016144.888843:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.888854:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016144.888865:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016144.888868:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e7464 00000080:00010000:0.0:1734016144.888872:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7464 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016144.888876:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.888882:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016144.888897:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.888909:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.888929:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.888934:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55e70, osc lock ffff8800a9f224c0, flags 0x400001002 00000008:00010000:0.0:1734016144.888941:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.888942:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55eb0, osc lock ffff8800a9f23300, flags 0x400001002 00010000:00010000:0.0:1734016144.888962:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e74b1 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.888970:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e74b1 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.889034:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e74b8 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.889040:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e74b8 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.894287:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.894326:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.894350:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016144.894361:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335434752 00000100:00100000:3.0:1734016144.894368:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335434752 00000100:00100000:3.0:1734016144.894390:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335434880 00000100:00100000:3.0:1734016144.894392:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335434880 00000100:00100000:3.0:1734016144.894401:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12783 00000100:00100000:3.0:1734016144.894406:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335434752:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.894418:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.894428:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.894448:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.894450:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 6127477 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 192 00000020:00010000:3.0:1734016144.894453:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016144.894454:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 6127477 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 192 00000100:00100000:3.0:1734016144.894505:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335434752:12345-192.168.201.116@tcp:106: Request processed in 99us (226us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.894510:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12783 00000100:00100000:3.0:1734016144.894520:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12784 00000100:00100000:3.0:1734016144.894524:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dfc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335434880:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016144.894530:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016144.894537:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016144.894551:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016144.894553:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2728624 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 0 00000020:00010000:3.0:1734016144.894556:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016144.894557:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2728624 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 0 00000100:00100000:3.0:1734016144.894572:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dfc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335434880:12345-192.168.201.116@tcp:106: Request processed in 47us (245us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016144.894575:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12784 00000080:00200000:0.0:1734016144.907351:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10321781 i_blocks 192 00000020:00010000:0.0:1734016144.907360:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cb908 (ffff8800a8a54a28/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.907362:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cb908 (ffff8800a8a54a28/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.907371:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55e70 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.907375:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55eb0 (ffff8800a8a541b8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.907376:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cb908 (ffff8800a8a54a28/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.907378:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55e70 (ffff8800a8a54a28/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.907382:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55eb0 (ffff8800a8a54a28/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.907395:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.907420:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016144.907425:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.907436:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.907459:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.907462:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55e70, osc lock ffff8800a9f23300, flags 0x400001002 00000008:00010000:0.0:1734016144.907651:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.907653:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55eb0, osc lock ffff8800a9f224c0, flags 0x400001002 00010000:00010000:0.0:1734016144.907672:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74bf lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.907680:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74bf lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.907732:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74c6 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.907739:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74c6 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.916748:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.917265:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016144.926080:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10321781 i_blocks 192 00000020:00010000:0.0:1734016144.926086:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cb908 (ffff8800a8a54a28/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.926089:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cb908 (ffff8800a8a54a28/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.926093:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55e70 (ffff8800a8a541b8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.926097:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55eb0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.926099:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cb908 (ffff8800a8a54a28/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.926100:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55e70 (ffff8800a8a54a28/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.926104:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55eb0 (ffff8800a8a54a28/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.926116:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.926125:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10321781 00000080:00200008:0.0:1734016144.926140:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016144.926145:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016144.926160:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016144.926173:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.926182:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016144.926192:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016144.926195:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e7464 00000080:00010000:0.0:1734016144.926199:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7464 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016144.926201:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016144.926206:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016144.926221:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.926230:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.926248:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.926251:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55e70, osc lock ffff8800a9f224c0, flags 0x400001002 00000008:00010000:0.0:1734016144.926255:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.926257:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55eb0, osc lock ffff8800a9f23300, flags 0x400001002 00010000:00010000:0.0:1734016144.926271:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e74cd lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.926282:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e74cd lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.926335:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e74d4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.926342:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e74d4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016144.934648:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.934747:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016144.947369:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10321781 i_blocks 192 00000020:00010000:0.0:1734016144.947377:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cb908 (ffff8800a8a54a28/0) at cl_lock_release():233 00000020:00010000:0.0:1734016144.947380:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cb908 (ffff8800a8a54a28/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.947385:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55e70 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.947389:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55eb0 (ffff8800a8a541b8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016144.947390:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cb908 (ffff8800a8a54a28/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.947393:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55e70 (ffff8800a8a54a28/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016144.947396:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55eb0 (ffff8800a8a54a28/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016144.947407:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016144.947425:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016144.947430:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016144.947439:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016144.947458:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.947461:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55e70, osc lock ffff8800a9f23300, flags 0x400001002 00000008:00010000:0.0:1734016144.947502:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016144.947505:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55eb0, osc lock ffff8800a9f224c0, flags 0x400001002 00010000:00010000:0.0:1734016144.947525:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.947537:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.947591:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.947602:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333998848:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.947678:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.947685:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.947713:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333998976:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.954982:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.955038:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016144.969052:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969083:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969096:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969103:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969109:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.969122:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2728624, setting rss=2728624; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.969129:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969135:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969141:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969148:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.969153:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969163:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.969168:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969174:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e74e2 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.969204:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333998976:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016144.969223:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969234:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969241:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969246:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969252:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.969259:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6127477, setting rss=6127477; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.969284:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969290:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969295:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969301:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c33c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.969303:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969314:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.969317:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.969323:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e74db lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.969337:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333998848:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016144.970220:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.970231:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333999104:192.168.201.116@tcp:22:fsx.0 00000100:00100000:0.0:1734016144.980672:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247333999104:192.168.201.116@tcp:22:fsx.0 00000100:00100000:0.0:1734016144.981528:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333999232:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016144.981653:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016144.981660:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333999360:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016144.986174:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016144.993309:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016144.995183:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.995196:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.995209:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.995220:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.995226:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016144.995240:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2728624, setting rss=2728624; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016144.995247:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.995252:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.995259:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.995265:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781ef40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016144.995271:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.995279:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016144.995285:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016144.995291:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781ef40/0xa50fd972cb8e74f0 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016144.995321:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333999360:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.002761:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.002781:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.002795:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.002801:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.002806:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.002819:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6127477, setting rss=6127477; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016145.002842:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.002847:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.002853:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.002862:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781cfc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.002868:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.002877:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016145.002880:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.002886:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e74e9 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.002909:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247333999232:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016145.011304:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016145.011358:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016145.042985:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.043002:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.043015:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.043022:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.043028:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.043042:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2728624, setting rss=2728624; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.043050:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.043055:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.043061:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.043067:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.043080:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.043091:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.043096:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.043102:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e750c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.043140:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333999872:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016145.050940:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.050954:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.050966:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.050970:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.050974:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.050987:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6127477, setting rss=6127477; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.051006:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.051011:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.051016:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.051020:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.051025:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.051034:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.051037:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.051040:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7505 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.051077:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247333999744:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016145.052165:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334000000:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016145.052901:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334000128:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016145.063142:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335436544 00000100:00100000:3.0:1734016145.063151:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335436544 00000100:00100000:3.0:1734016145.063197:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12797 00000100:00100000:3.0:1734016145.063203:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335436544:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.063218:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.063228:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.063253:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.063255:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 6127477 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 192 00000020:00010000:3.0:1734016145.063258:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016145.063259:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 6127477 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 192 00000100:00100000:3.0:1734016145.063303:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335436544:12345-192.168.201.116@tcp:106: Request processed in 99us (1418us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.063310:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12797 00000100:00100000:3.0:1734016145.069989:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016145.070011:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335436672 00000100:00100000:3.0:1734016145.070020:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335436672 00000100:00100000:3.0:1734016145.070440:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12798 00000100:00100000:3.0:1734016145.070446:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f858050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335436672:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.070458:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.070485:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.070512:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.070514:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2728624 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 16 00000020:00010000:3.0:1734016145.070516:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.070517:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2728624 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 16 00000100:00100000:3.0:1734016145.070655:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f858050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335436672:12345-192.168.201.116@tcp:106: Request processed in 208us (4170us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.070661:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12798 00010000:00010000:3.0:1734016145.088185:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.088201:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.088213:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.088218:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.088223:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.088235:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2728624, setting rss=2728624; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.088245:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.088254:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.088260:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.088265:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.088271:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.088280:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.088286:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.088292:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e751a lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.089971:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334000128:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016145.098945:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.098965:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.098977:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.098983:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.098987:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.098998:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6127477, setting rss=6127477; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.099003:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.099008:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.099014:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.099019:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.099024:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.099033:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.099038:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.099044:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e7521 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.101554:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334000256:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016145.101607:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.101618:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.101628:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.101634:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.101640:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.101651:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2728624, setting rss=2728624; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.101674:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.101680:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.101687:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.101697:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367cc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.101701:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.101711:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.101715:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.101721:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7528 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.101740:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334000384:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016145.109683:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335437056 00000100:00100000:3.0:1734016145.109690:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335437056 00000100:00100000:3.0:1734016145.109724:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12801 00000100:00100000:3.0:1734016145.109729:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335437056:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.109741:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.109751:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.109773:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.109775:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2728624 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 16 00000020:00010000:3.0:1734016145.109777:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.109779:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2728624 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 16 00000100:00100000:3.0:1734016145.109822:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335437056:12345-192.168.201.116@tcp:106: Request processed in 92us (312us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.109827:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12801 00000100:00100000:3.0:1734016145.114637:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335437184 00000100:00100000:3.0:1734016145.114646:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335437184 00000100:00100000:3.0:1734016145.114679:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12802 00000100:00100000:3.0:1734016145.114686:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335437184:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.114699:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.114709:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e74aa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 6123520->6127615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711786f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.114731:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.114733:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 6127477 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 192 00000020:00010000:3.0:1734016145.114736:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016145.114737:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 6127477 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 192 00000100:00100000:3.0:1734016145.114779:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335437184:12345-192.168.201.116@tcp:106: Request processed in 93us (267us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.114784:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12802 00010000:00010000:3.0:1734016145.118196:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.118215:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.118229:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.118236:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.118241:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.118256:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2728624, setting rss=2728624; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.118263:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.118267:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.118273:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.118279:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.118285:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.118297:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.118303:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.118309:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7536 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.118360:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334000640:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016145.125892:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10321781 i_blocks 208 00000020:00010000:0.0:1734016145.125903:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.125907:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.125912:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7f30 (ffff8800a8a54518/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.125916:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7f70 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.125918:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.125921:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7f30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.125927:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7f70 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.125941:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.125952:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10321781 00000080:00200000:0.0:1734016145.126004:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016145.126008:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016145.126020:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016145.126033:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [977, 979] 00000008:00010000:0.0:1734016145.126044:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.126047:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369991b0, osc lock ffff8800a9f22ab0, flags 0x0 00010000:00010000:0.0:1734016145.126083:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.126094:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4001792->4014079] (req 4001792->4014079) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.126127:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4001792->4014079] (req 4001792->4014079) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.126151:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334000768:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.126210:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55f00 going to sleep for 11 seconds 00000100:00100000:3.0:1734016145.131328:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016145.141529:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334000768:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.141558:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4001792->4014079] (req 4001792->4014079) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.141569:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4001792->4014079] (req 4001792->4014079) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.141576:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4001792->4014079] (req 4001792->4014079) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.141949:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335437440 00000100:00100000:3.0:1734016145.141956:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335437440 00000100:00100000:3.0:1734016145.141990:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12804 00000100:00100000:3.0:1734016145.141996:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335437440:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016145.142005:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4001792->4014079] (req 4001792->4014079) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.142012:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4001792->4014079] (req 4001792->4014079) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.142019:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.142026:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.142032:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.142040:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.142057:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c21c0) 00000100:00100000:3.0:1734016145.142109:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7698:x1818247335437440:12345-192.168.201.116@tcp:105: Request processed in 112us (779us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.142114:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12804 00010000:00010000:0.0:1734016145.142591:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.142600:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.142627:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.142639:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 6127477, setting rss=6127477; kms=6127477, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.142650:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.142657:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.142681:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10321781 i_blocks 208 00000100:00100000:0.0:1734016145.149521:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016145.149549:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334001024:192.168.201.116@tcp:22:fsx.0 00000100:00100000:0.0:1734016145.152351:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334001024:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016145.155373:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016145.155388:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335437568 00000100:00100000:3.0:1734016145.155396:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335437568 00000100:00100000:3.0:1734016145.155433:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12805 00000100:00100000:3.0:1734016145.155440:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2ddc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335437568:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.155453:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.155530:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.155555:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.155557:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2728624 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 16 00000020:00010000:3.0:1734016145.155560:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.155561:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2728624 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 16 00000100:00100000:3.0:1734016145.155604:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2ddc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335437568:12345-192.168.201.116@tcp:106: Request processed in 164us (348us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.155612:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12805 00010000:00010000:3.0:1734016145.160242:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.160258:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.160270:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.160275:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.160281:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.160292:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2728624, setting rss=2728624; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.160311:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.160317:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.160323:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.160328:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.160333:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.160342:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.160346:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.160352:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e7544 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.162159:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334001152:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016145.172763:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335437696 00000100:00100000:3.0:1734016145.172776:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335437696 00000100:00100000:3.0:1734016145.172833:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12806 00000100:00100000:3.0:1734016145.172839:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335437696:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.172853:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.172863:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.172888:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.172890:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2728624 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 16 00000020:00010000:3.0:1734016145.172893:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.172894:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2728624 mtime: 1734016144 atime: 1734016143 ctime: 1734016144 blocks: 16 00000100:00100000:3.0:1734016145.172944:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335437696:12345-192.168.201.116@tcp:106: Request processed in 105us (2960us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.172951:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12806 00010000:00010000:3.0:1734016145.190244:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.190261:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.190273:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.190279:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.190284:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.190296:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2728624, setting rss=2728624; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.190316:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.190321:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.190327:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.190332:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.190337:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.190346:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.190350:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.190355:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e754b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.190392:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334001280:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016145.197124:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10321781 i_blocks 208 00000020:00010000:0.0:1734016145.197131:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.197133:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.197138:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7e70 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.197141:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7eb0 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.197143:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.197145:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7e70 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.197148:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7eb0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.197159:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016145.197203:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 138 00000080:00200000:0.0:1734016145.197210:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016145.197224:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016145.197235:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [138, 138] 00000008:00010000:0.0:1734016145.197248:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.197251:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999cb0, osc lock ffff8800a9f22f70, flags 0x0 00010000:00010000:0.0:1734016145.197270:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.197277:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [565248->569343] (req 565248->569343) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.197301:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [565248->569343] (req 565248->569343) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.197318:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334001536:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.197366:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55b40 going to sleep for 11 seconds 00000100:00100000:3.0:1734016145.199372:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335437952 00000100:00100000:3.0:1734016145.199381:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335437952 00000100:00100000:3.0:1734016145.199427:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12808 00000100:00100000:3.0:1734016145.199434:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335437952:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016145.199445:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.199519:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016145.199524:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335437952:12345-192.168.201.116@tcp:104: Request processed in 91us (237us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.199530:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12808 00010000:00010000:0.0:1734016145.199687:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016145.199694:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.199700:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c21c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.199705:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.199724:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016145.199735:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016145.199766:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 2/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.199772:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016145.199796:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.199802:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e753d lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4001792->4014079) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117902 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.199824:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.199825:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.199826:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016145.204088:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335438080 00000100:00100000:3.0:1734016145.204097:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335438080 00000100:00100000:3.0:1734016145.204141:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12809 00000100:00100000:3.0:1734016145.204147:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7698:x1818247335438080:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016145.204160:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [565248->569343] (req 565248->569343) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.204170:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [565248->569343] (req 565248->569343) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.204177:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.204192:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.204200:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.204209:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.204226:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c2400) 00000100:00100000:3.0:1734016145.204278:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7698:x1818247335438080:12345-192.168.201.116@tcp:105: Request processed in 130us (658us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.204284:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12809 00000100:00100000:0.0:1734016145.208102:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334001536:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.208127:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.208151:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.208163:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 6127477, setting rss=6127477; kms=6127477, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.208174:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.208179:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.208258:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.208267:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.208274:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.208279:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.208283:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.211975:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.211991:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016145.238787:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.238790:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.238794:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999cb0 (ffff88012dc12368/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016145.238799:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.238808:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016145.238821:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.238823:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999cb0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.238840:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016145.238876:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 139 00000080:00200000:0.0:1734016145.238882:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016145.238897:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016145.238909:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [139, 139] 00000008:00010000:0.0:1734016145.238921:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.238925:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999cb0, osc lock ffff8800a9f22f70, flags 0x0 00010000:00010000:0.0:1734016145.238940:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.238950:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (569344 573439) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.238958:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.238963:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.238968:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.239005:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.239012:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2732032 2736127) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.239018:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.239023:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.239027:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.239033:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.239038:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.291544:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016145.296724:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334002816:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016145.299893:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016145.303791:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10321781 i_blocks 208 00000020:00010000:0.0:1734016145.303799:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.303802:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.303808:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7cf0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.303813:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7d30 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.303816:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.303818:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.303821:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.303833:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.303844:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10321781 00000080:00200008:0.0:1734016145.303909:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 6598310, type: SEEK_SET 00000080:00200008:0.0:1734016145.303912:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 6598310, type: SEEK_SET, rc = 6598310 00000080:00200008:0.0:1734016145.303921:0:10406:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 6598310, count: 3241 00000080:00200000:0.0:1734016145.303926:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016145.303937:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016145.303966:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 1610 00000080:00200000:0.0:1734016145.303973:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 6598310, bytes: 3241 00000080:00200000:0.0:1734016145.303976:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016145.303981:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016145.303983:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [6598310, 6601551) 0 0 00000080:00200000:0.0:1734016145.303985:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016145.303998:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [6598310, 6601551) 6601551, 3241 00020000:00200000:0.0:1734016145.304003:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016145.304019:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [1610, 1611] 00000008:00010000:0.0:1734016145.304039:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.304042:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369990b0, osc lock ffff8800a9f23a20, flags 0x0 00010000:00010000:0.0:1734016145.304075:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.304083:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [2400256->2408447] (req 2400256->2408447) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.304116:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [2400256->2408447] (req 2400256->2408447) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.304139:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334002944:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.305188:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55180 going to sleep for 11 seconds 00000100:00100000:3.0:1734016145.310941:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016145.312132:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016145.312140:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.312146:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3a80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.312152:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.312176:0:4750:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016145.314489:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016145.314540:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 2/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.314548:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016145.326802:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.326817:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e745d lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117814 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.326855:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.326856:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.326860:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016145.328277:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334002944:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016145.328295:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.328313:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.328328:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 2728624, setting rss=2728624; kms=2728624, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.328339:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.328343:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016145.328354:0:10406:0:(vvp_io.c:840:vvp_io_read_start()) f16b.sanityn: read [6598310, 6601551) 00000080:00200000:3.0:1734016145.328379:0:10406:0:(vvp_io.c:899:vvp_io_read_start()) count:3241 iocb pos:6598310 00010000:00010000:3.0:1734016145.328423:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.328432:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.339299:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.339313:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.339347:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.339362:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.342146:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.342158:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.422043:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8801369d4cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.422260:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8801369d4cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7575 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2400256->2408447) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711793a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.422292:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334003328:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016145.468375:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016145.494052:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016145.495517:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334003712:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.495535:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.495555:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.495565:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 2728624, setting rss=2728624; kms=2728624, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.495576:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.495580:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.495593:0:10406:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [7156154, 7161950) 00000080:00200000:0.0:1734016145.495624:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 1747 of 442 to 3654 bytes 00010000:00010000:0.0:1734016145.495675:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.495684:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2732032 2736127) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.495690:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 7/1,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.495695:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 7/1,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.495701:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.495709:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.495712:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.495758:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:0.0:1734016145.495773:0:1982:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a550c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.495798:0:1982:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2961408 2965503) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016145.495828:0:1982:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a934ce00 - object 0x280000400:8 offset 2961408<>2965504 00000100:00100000:0.0:1734016145.495843:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334004096:192.168.201.116@tcp:3:fsx.0 00000080:00200000:0.0:1734016145.506211:0:10406:0:(rw26.c:1006:ll_write_end()) pos 7156154, len 3654, copied 3654 00000080:00200000:0.0:1734016145.506220:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800b39f20d8[3 ffff8800b3a5ce90 2 1 ffff8800a79c9058] 00000080:00200000:0.0:1734016145.506223:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a5e740 1fffff0000080d 3:0 ffff8800b39f20d8 1747 lru 00000080:00200000:0.0:1734016145.506228:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800b39f2138 723: 1< 1 - - > 2< 2961408 0 4096 0x0 0x40009 | ffff8800a9d60708 ffff8800aa37b1e0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016145.506230:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800b39f20d8 00000080:00200000:0.0:1734016145.506231:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:0.0:1734016145.506234:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 1748 of 0 to 2142 bytes 00010000:00010000:0.0:1734016145.506295:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.506306:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2732032 2736127) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.506316:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 7/1,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.506322:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 7/1,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.506327:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.506335:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.506341:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.511686:0:10406:0:(rw26.c:1006:ll_write_end()) pos 7159808, len 2142, copied 2142 00000080:00200000:0.0:1734016145.511693:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800b39f2000[3 ffff8800b3a5ce90 2 1 ffff8800a79c9058] 00000080:00200000:0.0:1734016145.511696:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a007c0 1fffff0000080d 3:0 ffff8800b39f2000 1748 lru 00000080:00200000:0.0:1734016145.511702:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800b39f2060 724: 1< 1 - - > 2< 2965504 0 4096 0x0 0x40009 | ffff8800a9d60708 ffff8800aa37b1e0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016145.511703:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800b39f2000 00000080:00200000:0.0:1734016145.511704:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:0.0:1734016145.511713:0:10406:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 2, from 442, to 2142 00000008:00010000:0.0:1734016145.511741:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800a8a55840 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.511765:0:10406:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800a8a55840 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.511774:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134a48, count 2, dirtied 2 00000080:00200000:0.0:1734016145.511779:0:10406:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 2 pages 5796 bytes, tot: 5796 00000080:00200000:0.0:1734016145.511781:0:10406:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 5796 00000020:00010000:0.0:1734016145.511794:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.511797:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.511801:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811ab0 (ffff88012dc12758/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016145.511809:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.511817:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016145.511833:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.511835:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811ab0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.511841:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 5796 bytes 00000080:00200000:0.0:1734016145.511846:0:10406:0:(file.c:1930:ll_file_io_generic()) Range unlock [1747, 1748] 00000080:00200000:0.0:1734016145.511853:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016145.511856:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 5796, restart: 0 00000080:00200000:0.0:1734016145.511868:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10321781 i_blocks 208 00000080:00200000:0.0:1734016145.511873:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 5796 00000080:00200008:0.0:1734016145.511891:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016145.511896:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016145.511911:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016145.511924:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.511935:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016145.511944:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016145.511947:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e7464 00000080:00010000:0.0:1734016145.511951:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7464 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016145.511953:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.511959:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016145.511971:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.511984:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.512002:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.512005:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23b50, flags 0x400001002 00000008:00010000:0.0:1734016145.512011:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.512012:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23560, flags 0x400001002 00010000:00010000:0.0:1734016145.512023:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.512030:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.512038:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.512048:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.512076:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1d40/0xa50fd972cb8e7583 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.512083:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1d40/0xa50fd972cb8e7583 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.518395:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335438976 00000100:00100000:3.0:1734016145.518403:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335438976 00000100:00100000:3.0:1734016145.518443:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12816 00000100:00100000:3.0:1734016145.518449:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335438976:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.518491:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.518505:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.518528:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 5, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.518529:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2967646 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000020:00010000:3.0:1734016145.518532:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.518533:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2967646 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000100:00100000:3.0:1734016145.518570:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335438976:12345-192.168.201.116@tcp:106: Request processed in 121us (278us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.518576:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12816 00000080:00200000:0.0:1734016145.522431:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10321781 i_blocks 208 00000020:00010000:0.0:1734016145.522438:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.522441:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.522446:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.522450:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.522452:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.522455:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.522458:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.522489:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.522505:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016145.522510:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.522520:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.522543:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.522546:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23560, flags 0x400001002 00000008:00010000:0.0:1734016145.522552:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.522553:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23b50, flags 0x400001002 00010000:00010000:0.0:1734016145.522568:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.522578:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.522585:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.522591:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.522614:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e758a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.522621:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1f80/0xa50fd972cb8e758a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.526708:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335439104 00000100:00100000:3.0:1734016145.526716:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335439104 00000100:00100000:3.0:1734016145.526809:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12817 00000100:00100000:3.0:1734016145.526815:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335439104:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.526828:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.526837:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.526859:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 5, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.526861:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2967646 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000020:00010000:3.0:1734016145.526864:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.526865:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2967646 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000100:00100000:3.0:1734016145.526905:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335439104:12345-192.168.201.116@tcp:106: Request processed in 90us (282us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.526911:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12817 00000080:00200000:0.0:1734016145.530296:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10321781 i_blocks 208 00000020:00010000:0.0:1734016145.530302:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.530305:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.530312:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.530317:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.530319:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.530321:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.530325:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.530337:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.530345:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10321781 00000080:00200008:0.0:1734016145.530421:0:10406:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8) current size 10321781, valid attrs 2068, mode 8801, uid -1227447776, gid -30720, new size 8344329, atime 1734016145.000000000, mtime 1734016145.000000000, ctime 1734016145.000000000 00010000:00010000:0.0:1734016145.530444:0:10406:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.530461:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.530519:0:10406:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.530524:0:10406:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016145.530526:0:10406:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e7464 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27117822 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000020:00100000:0.0:1734016145.530551:0:10406:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff88012a571000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1734016145.530562:0:10406:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a934f800 x1818247334004864/t0(0) o36->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016145.530582:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334004864:192.168.201.116@tcp:36:fsx.0 00000100:00100000:0.0:1734016145.531273:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1734016145.535550:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016145.535566:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335439232 00000100:00100000:3.0:1734016145.535573:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335439232 00000100:00100000:3.0:1734016145.535615:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12818 00000100:00100000:3.0:1734016145.535621:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88013770bc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6479:x1818247335439232:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016145.535636:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016145.535698:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016145.535704:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88013770bc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6479:x1818247335439232:12345-192.168.201.116@tcp:104: Request processed in 82us (323us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.535710:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12818 00010000:00010000:0.0:1734016145.535876:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016145.535884:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.535890:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa10571e0) 00010000:00010000:0.0:1734016145.535894:0:5248:0:(ldlm_request.c:1579:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.535900:0:5248:0:(ldlm_request.c:1589:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.535907:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.535926:0:5248:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781c900/0xa50fd972cb8e5f33 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115fb0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.535960:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 2/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.535964:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016145.535990:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.535995:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e749c lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117861 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.536012:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.536013:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.536015:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016145.544515:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334004864:192.168.201.116@tcp:36:fsx.0 00000080:00200000:0.0:1734016145.544562:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016145.544567:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016145.544585:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016145.544607:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1734016145.544623:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.544627:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23b50, flags 0x0 00000008:00010000:0.0:1734016145.544630:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.544631:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23560, flags 0x0 00010000:00010000:0.0:1734016145.544648:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800ac554800 type 11 mode 2 res 9663677440/6 (0 18446744073709551615) 00010000:00010000:0.0:1734016145.544661:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.544671:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.544702:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.544719:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334005120:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.544751:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55540 going to sleep for 11 seconds 00000100:00100000:3.0:1734016145.549443:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016145.549873:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016145.549896:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335439360 00000100:00100000:3.0:1734016145.549903:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335439360 00000100:00100000:3.0:1734016145.550030:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335439488 00000100:00100000:3.0:1734016145.550033:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335439488 00010000:00010000:0.0:1734016145.550844:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1734016145.550851:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.550858:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781c240 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.550864:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.550881:0:4750:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016145.554455:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016145.554532:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.554542:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016145.554569:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.554580:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e7560 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117925 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.554595:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016145.554598:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.554604:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c2400 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.554608:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.554620:0:4750:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016145.554662:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016145.554679:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.554684:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016145.554689:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.554694:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7559 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 565248->569343) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc2711791e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.554707:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.554708:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.554711:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016145.566575:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016145.568037:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.568048:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.568078:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.568089:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 6127477, setting rss=6127477; kms=6127477, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.568099:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.568104:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.568128:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.568133:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.568157:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.568170:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0380 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334005504:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.568205:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55300 going to sleep for 11 seconds 00000100:00100000:3.0:1734016145.575667:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016145.576028:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016145.576035:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.576042:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.576047:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.576080:0:5248:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55840 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.576096:0:5248:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800a8a55840 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.576139:0:5248:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2961408 2965503) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016145.576205:0:5248:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e0e00 - object 0x280000400:8 offset 2961408<>2967646 00000100:00100000:0.0:1734016145.580046:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0380 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334005504:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.580080:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.580090:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.580097:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.583292:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016145.583421:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016145.583485:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 2/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.583492:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016145.583521:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.583527:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e757c lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2961408->2969599) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117941 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.583545:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.583546:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.583548:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016145.588796:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016145.590389:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.590410:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.590432:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.590444:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 2967646, setting rss=2967646; kms=2967646, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.590459:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.590482:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.597354:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 216 00000020:00010000:0.0:1734016145.597364:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.597366:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.597370:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff88012dc12758/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016145.597382:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.597393:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016145.597405:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016145.597409:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.597415:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016145.597421:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.597423:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.597428:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.597445:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.597461:0:10406:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8) current size 8344329, valid attrs 2168, mode 8801, uid -1227447776, gid -30720, new size 8344329, atime 1734016145.000000000, mtime 1734016145.529000000, ctime 1734016145.529000000, rc 0 00000080:00200008:0.0:1734016145.597509:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016145.597514:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016145.597529:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016145.597540:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016145.597577:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016145.597583:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016145.597591:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016145.597608:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1f80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334006144:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.597877:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1734016145.601960:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1f80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334006144:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.601979:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.601988:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.602001:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.602012:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.602021:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.602027:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.602032:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016145.602039:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78e1f80 x1818247334006144/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016161 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016145.602046:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016145.602056:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.602073:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016145.602124:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016145.602129:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e759f 00000080:00010000:0.0:1734016145.602132:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016145.602136:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e759f from it ffff8801369fbcd8 00010000:00010000:0.0:1734016145.602138:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.602149:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016145.602175:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.602185:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.602205:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602208:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23560, flags 0x400001002 00000008:00010000:0.0:1734016145.602213:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602215:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23b50, flags 0x400001002 00010000:00010000:0.0:1734016145.602230:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602239:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602249:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602254:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602267:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602274:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602284:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602290:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.602303:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 216 00000020:00010000:0.0:1734016145.602308:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.602310:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602312:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602314:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602316:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602318:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602321:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.602328:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.602341:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016145.602345:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.602347:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.602351:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602352:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23b50, flags 0x400001002 00000008:00010000:0.0:1734016145.602354:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602355:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23560, flags 0x400001002 00010000:00010000:0.0:1734016145.602360:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602365:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602372:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602378:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602386:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602392:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602398:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602403:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.602411:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 216 00000020:00010000:0.0:1734016145.602413:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.602414:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602416:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602418:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602420:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602422:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602424:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.602428:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.602432:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8344329 00000080:00200008:0.0:1734016145.602442:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016145.602446:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016145.602456:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016145.602462:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.602501:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016145.602507:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016145.602509:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e759f 00000080:00010000:0.0:1734016145.602511:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e759f from it ffff8801369fbcd8 00010000:00010000:0.0:1734016145.602514:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.602519:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016145.602526:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.602529:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.602532:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602533:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23560, flags 0x400001002 00000008:00010000:0.0:1734016145.602536:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602537:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23b50, flags 0x400001002 00010000:00010000:0.0:1734016145.602541:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602550:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602556:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602561:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602567:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602572:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602577:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602582:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.602588:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 216 00000020:00010000:0.0:1734016145.602590:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.602592:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602593:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602596:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602597:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602599:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602601:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.602605:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.602609:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016145.602612:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.602614:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.602616:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602618:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23b50, flags 0x400001002 00000008:00010000:0.0:1734016145.602620:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602621:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23560, flags 0x400001002 00010000:00010000:0.0:1734016145.602624:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602630:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602635:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602640:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602647:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602651:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602657:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602662:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.602667:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 216 00000020:00010000:0.0:1734016145.602669:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.602670:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602672:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602673:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602675:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602676:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602678:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.602681:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.602683:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8344329 00000080:00200008:0.0:1734016145.602687:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016145.602689:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016145.602692:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016145.602696:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.602701:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016145.602708:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016145.602710:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e759f 00000080:00010000:0.0:1734016145.602713:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e759f from it ffff8801369fbcd8 00010000:00010000:0.0:1734016145.602714:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.602719:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016145.602725:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.602726:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.602729:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602730:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23560, flags 0x400001002 00000008:00010000:0.0:1734016145.602732:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602732:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23b50, flags 0x400001002 00010000:00010000:0.0:1734016145.602736:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602740:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602746:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602751:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602761:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602767:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602772:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602777:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.602782:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 216 00000020:00010000:0.0:1734016145.602784:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.602785:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602786:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602788:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602789:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602791:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602792:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.602795:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.602798:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016145.602800:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.602802:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.602804:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602805:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23b50, flags 0x400001002 00000008:00010000:0.0:1734016145.602807:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602808:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23560, flags 0x400001002 00010000:00010000:0.0:1734016145.602811:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602816:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602821:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602826:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602832:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602836:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602841:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602846:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.602851:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 216 00000020:00010000:0.0:1734016145.602853:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.602855:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602856:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602858:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602859:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602861:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602862:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.602865:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.602867:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8344329 00000080:00200008:0.0:1734016145.602872:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016145.602874:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016145.602877:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016145.602880:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.602884:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016145.602889:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016145.602891:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e759f 00000080:00010000:0.0:1734016145.602893:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e759f from it ffff8801369fbcd8 00010000:00010000:0.0:1734016145.602895:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.602900:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016145.602905:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.602908:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.602911:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602912:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23560, flags 0x400001002 00000008:00010000:0.0:1734016145.602914:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.602915:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23b50, flags 0x400001002 00010000:00010000:0.0:1734016145.602918:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602924:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602930:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602936:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602945:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602951:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602957:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.602963:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.602973:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 216 00000020:00010000:0.0:1734016145.602975:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.602977:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602979:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602981:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.602982:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602984:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.602986:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.602989:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.602993:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016145.602995:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.602997:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.603000:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.603001:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23b50, flags 0x400001002 00000008:00010000:0.0:1734016145.603003:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.603004:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23560, flags 0x400001002 00010000:00010000:0.0:1734016145.603007:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603013:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603018:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603023:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603030:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603035:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603040:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603045:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.603050:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 216 00000020:00010000:0.0:1734016145.603051:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.603053:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.603054:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.603056:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.603057:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.603059:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.603071:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.603075:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.603078:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8344329 00000080:00200008:0.0:1734016145.603125:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 1042417, type: SEEK_SET 00000080:00200008:0.0:1734016145.603128:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 1042417, type: SEEK_SET, rc = 1042417 00000080:00200008:0.0:1734016145.603134:0:10406:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 1042417, count: 3136 00000080:00200000:0.0:1734016145.603140:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 1042417, bytes: 3136 00000080:00200000:0.0:1734016145.603143:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016145.603147:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016145.603150:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [1042417, 1045553) 0 0 00000080:00200000:0.0:1734016145.603152:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016145.603159:0:10406:0:(file.c:1893:ll_file_io_generic()) Range lock [254, 255] 00020000:00200000:0.0:1734016145.603166:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [1042417, 1045553) 1045553, 3136 00020000:00200000:0.0:1734016145.603171:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016145.603182:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [254, 255] 00000008:00010000:0.0:1734016145.603188:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.603190:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999a30, osc lock ffff8800a9f23560, flags 0x0 00010000:00010000:0.0:1734016145.603196:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603203:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1040384 1048575) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603209:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603214:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603219:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.603227:0:10406:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [1042417, 1045553) 00000080:00200000:0.0:1734016145.603251:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 254 of 2033 to 2063 bytes 00010000:00010000:0.0:1734016145.603295:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603306:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2732032 2736127) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603312:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603316:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.603321:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.628450:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.628493:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.628518:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334006912:192.168.201.116@tcp:3:fsx.0 00000020:00010000:0.0:1734016145.642018:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.642022:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.642027:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811db0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016145.642036:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.642047:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016145.642058:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.642069:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811db0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.642083:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.642128:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016145.642135:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016145.642150:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016145.642161:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016145.642191:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016145.642197:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016145.642205:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016145.642221:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1f80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334007168:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.642263:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1734016145.644632:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1f80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334007168:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.644656:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.644664:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.644676:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.644684:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.644692:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.644701:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.644707:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016145.644716:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78e1f80 x1818247334007168/t0(0) o101->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016161 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016145.644725:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016145.644734:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.644745:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016145.644798:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016145.644802:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e75a6 00000080:00010000:0.0:1734016145.644807:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016145.644810:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e75a6 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016145.644812:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.644819:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016145.644849:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.644860:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.644881:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.644885:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f23b50, flags 0x400001002 00000008:00010000:0.0:1734016145.644891:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.644892:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f238f0, flags 0x400001002 00010000:00010000:0.0:1734016145.644912:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e75ad lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.644918:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e75ad lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.647290:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75b4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.647301:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75b4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.662480:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016145.662501:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335440000 00000100:00100000:3.0:1734016145.662508:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335440000 00000100:00100000:3.0:1734016145.662558:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12824 00000100:00100000:3.0:1734016145.662564:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dd850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335440000:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.662576:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.662586:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.662607:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 350, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.662609:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000020:00010000:3.0:1734016145.662612:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016145.662613:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000100:00100000:3.0:1734016145.662657:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dd850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335440000:12345-192.168.201.116@tcp:106: Request processed in 92us (2714us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.662662:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12824 00000100:00100000:3.0:1734016145.664181:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335440128 00000100:00100000:3.0:1734016145.664187:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335440128 00000100:00100000:3.0:1734016145.664210:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12825 00000100:00100000:3.0:1734016145.664216:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335440128:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.664226:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.664235:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.664253:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 350, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.664254:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4150025 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 32 00000020:00010000:3.0:1734016145.664257:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.664257:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4150025 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 32 00000100:00100000:3.0:1734016145.664287:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335440128:12345-192.168.201.116@tcp:106: Request processed in 72us (181us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.664293:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12825 00000080:00200000:0.0:1734016145.669996:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 232 00000020:00010000:0.0:1734016145.670002:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.670005:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.670011:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.670014:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff8800a8a54518/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.670016:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.670019:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.670022:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.670033:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.670051:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016145.670056:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.670073:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.670087:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.670090:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55cf0, osc lock ffff8800a9f238f0, flags 0x400001002 00000008:00010000:0.0:1734016145.670093:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.670094:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55d30, osc lock ffff8800a9f23b50, flags 0x400001002 00010000:00010000:0.0:1734016145.670112:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e75bb lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.670118:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e75bb lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.670161:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75c2 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.670168:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75c2 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.676386:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016145.676630:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016145.676655:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335440256 00000100:00100000:3.0:1734016145.676666:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335440256 00000100:00100000:3.0:1734016145.676699:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335440384 00000100:00100000:3.0:1734016145.676701:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335440384 00000100:00100000:3.0:1734016145.676711:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12826 00000100:00100000:3.0:1734016145.676718:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7982d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335440256:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.676732:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.676742:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.676766:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 350, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.676769:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000020:00010000:3.0:1734016145.676773:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016145.676774:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000100:00100000:3.0:1734016145.676815:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7982d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335440256:12345-192.168.201.116@tcp:106: Request processed in 97us (440us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.676821:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12826 00000100:00100000:3.0:1734016145.676832:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12827 00000100:00100000:3.0:1734016145.676836:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335440384:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.676842:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.676849:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.676860:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 350, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.676861:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4150025 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 32 00000020:00010000:3.0:1734016145.676863:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.676865:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4150025 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 32 00000100:00100000:3.0:1734016145.676878:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335440384:12345-192.168.201.116@tcp:106: Request processed in 41us (249us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.676882:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12827 00000080:00200000:0.0:1734016145.682851:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 232 00000020:00010000:0.0:1734016145.682858:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.682860:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.682865:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55cf0 (ffff8800a8a54518/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.682868:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55d30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.682870:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.682872:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55cf0 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.682876:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55d30 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.682886:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.682896:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8344329 00000080:00200008:0.0:1734016145.682946:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 3843701, type: SEEK_SET 00000080:00200008:0.0:1734016145.682951:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 3843701, type: SEEK_SET, rc = 3843701 00000080:00200008:0.0:1734016145.682958:0:10406:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 3843701, count: 1731 00000080:00200000:0.0:1734016145.682966:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 938 of 1653 to 1731 bytes 00000080:00200000:0.0:1734016145.682971:0:10406:0:(file.c:2441:ll_do_tiny_write()) result: 0, original count 1731 00000080:00200000:0.0:1734016145.682974:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 3843701, bytes: 1731 00000080:00200000:0.0:1734016145.682977:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016145.682982:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5cd48[0x4, 350, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016145.682984:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [3843701, 3845432) 0 0 00000080:00200000:0.0:1734016145.682987:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016145.683002:0:10406:0:(file.c:1893:ll_file_io_generic()) Range lock [938, 938] 00020000:00200000:0.0:1734016145.683008:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [3843701, 3845432) 3845432, 1731 00020000:00200000:0.0:1734016145.683012:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016145.683027:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [938, 938] 00000008:00010000:0.0:1734016145.683040:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.683043:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999030, osc lock ffff8800a9f23b50, flags 0x0 00010000:00010000:0.0:1734016145.683057:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.683076:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3842048 3846143) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.683083:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.683089:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.683095:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.683101:0:10406:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [3843701, 3845432) 00000080:00200000:0.0:1734016145.683109:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 938 of 1653 to 1731 bytes 00000080:00200000:0.0:1734016145.688426:0:10406:0:(rw26.c:1006:ll_write_end()) pos 3843701, len 1731, copied 1731 00000080:00200000:0.0:1734016145.688433:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800b39f2d80[3 ffff8800b3a5cdc8 2 1 ffff8800a79c9058] 00000080:00200000:0.0:1734016145.688436:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a01540 1fffff0000080d 3:0 ffff8800b39f2d80 938 lru 00000080:00200000:0.0:1734016145.688443:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800b39f2de0 938: 1< 1 - - > 2< 3842048 0 4096 0x0 0x40009 | ffff88012a41dcc8 ffff8800a9006000 > 3< 0 0 > 4< 0 0 8 489848832 - | - - + - > 5< - - + - | 0 - | 2 - -> 00000080:00200000:0.0:1734016145.688445:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800b39f2d80 00000080:00200000:0.0:1734016145.688446:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:0.0:1734016145.688451:0:10406:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 1, from 1653, to 3384 00000008:00010000:0.0:1734016145.688501:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800a8a55cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.688516:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134248, count 1, dirtied 1 00000080:00200000:0.0:1734016145.688520:0:10406:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 1 pages 1731 bytes, tot: 1731 00000080:00200000:0.0:1734016145.688523:0:10406:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 1731 00000020:00010000:0.0:1734016145.688536:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.688538:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.688542:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999030 (ffff8800a8a54128/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016145.688549:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.688557:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016145.688569:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a326c8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.688571:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999030 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.688577:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 1731 bytes 00000080:00200000:0.0:1734016145.688581:0:10406:0:(file.c:1930:ll_file_io_generic()) Range unlock [938, 938] 00000080:00200000:0.0:1734016145.688589:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016145.688592:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 1731, restart: 0 00000080:00200000:0.0:1734016145.688604:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 232 00000080:00200000:0.0:1734016145.688608:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 1731 00000080:00200008:0.0:1734016145.688626:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016145.688631:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016145.688645:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016145.688657:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.688666:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016145.688676:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016145.688678:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e75a6 00000080:00010000:0.0:1734016145.688683:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e75a6 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016145.688686:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.688691:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016145.688703:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.688711:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.688722:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.688724:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55030, osc lock ffff8800a9f23b50, flags 0x400001002 00000008:00010000:0.0:1734016145.688729:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.688730:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55070, osc lock ffff8800a9f238f0, flags 0x400001002 00010000:00010000:0.0:1734016145.688746:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e75c9 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.688753:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1680/0xa50fd972cb8e75c9 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.688792:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e75d0 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.688798:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e75d0 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.694022:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016145.699344:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 232 00000020:00010000:0.0:1734016145.699351:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.699353:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.699359:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55030 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.699363:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55070 (ffff8800a8a54518/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.699364:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.699367:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55030 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.699370:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55070 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.699380:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.699402:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016145.699407:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.699417:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.699433:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.699435:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55030, osc lock ffff8800a9f238f0, flags 0x400001002 00000008:00010000:0.0:1734016145.699438:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.699440:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55070, osc lock ffff8800a9f23b50, flags 0x400001002 00010000:00010000:0.0:1734016145.699459:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.700209:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.700254:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e75de lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.700264:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e75de lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.700734:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334008320:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.710130:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.710145:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.710156:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.710163:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.710168:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.710180:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016145.710186:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.710192:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.710203:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.710209:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c06c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.710215:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.710224:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016145.710228:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.710234:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75d7 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.710263:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334008320:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016145.710839:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8344329 i_blocks 232 00000020:00010000:0.0:1734016145.710844:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.710847:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.710852:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55030 (ffff8800a8a54518/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.710855:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55070 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.710857:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8ff8 (ffff8800a8a541b8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.710860:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55030 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.710863:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55070 (ffff8800a8a541b8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.710874:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.710884:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8344329 00000080:00200008:0.0:1734016145.710941:0:10406:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8) current size 8344329, valid attrs 2068, mode 8801, uid -1227447776, gid -30720, new size 6363171, atime 1734016145.000000000, mtime 1734016145.000000000, ctime 1734016145.000000000 00010000:00010000:0.0:1734016145.710968:0:10406:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.710984:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.711017:0:10406:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.711022:0:10406:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016145.711024:0:10406:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e759f lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27117972 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000020:00100000:0.0:1734016145.711037:0:10406:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff88012a571000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1734016145.711045:0:10406:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a78e3800 x1818247334008704/t0(0) o36->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016145.711071:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3800 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334008704:192.168.201.116@tcp:36:fsx.0 00000100:00100000:0.0:1734016145.711123:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55000 going to sleep for 11 seconds 00000100:00100000:3.0:1734016145.714196:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335441024 00000100:00100000:3.0:1734016145.714205:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335441024 00000100:00100000:3.0:1734016145.714385:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12832 00000100:00100000:3.0:1734016145.714560:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6479:x1818247335441024:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016145.714573:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e75a6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117980 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016145.714674:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016145.714680:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6479:x1818247335441024:12345-192.168.201.116@tcp:104: Request processed in 289us (957us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.714687:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12832 00000100:00100000:0.0:1734016145.719539:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3800 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334008704:192.168.201.116@tcp:36:fsx.0 00000080:00200000:0.0:1734016145.719572:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016145.719576:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016145.719594:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016145.719611:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1734016145.719624:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.719627:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55030, osc lock ffff8800a9f23b50, flags 0x0 00000008:00010000:0.0:1734016145.719630:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.719631:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55070, osc lock ffff8800a9f238f0, flags 0x0 00010000:00010000:0.0:1734016145.719648:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.719659:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.719668:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 8/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.719674:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 8/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.719680:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 7/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.719689:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.719694:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.719699:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.719704:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.719714:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.719801:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334009088:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016145.730733:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334009088:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016145.736097:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334009216:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.736123:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.736133:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.736146:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.736155:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.736163:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.736169:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.736174:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016145.736182:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78a1f80 x1818247334009216/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016161 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016145.736192:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016145.736201:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.736212:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016145.736260:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016145.736265:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e75e5 00000080:00010000:0.0:1734016145.736270:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016145.736272:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e75e5 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016145.736274:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.736281:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016145.736310:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.736320:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.736366:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.736370:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55330, osc lock ffff8800a9f231d0, flags 0x400001002 00000008:00010000:0.0:1734016145.736387:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.736388:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55370, osc lock ffff8800a9f23690, flags 0x400001002 00010000:00010000:0.0:1734016145.736690:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736707:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736717:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 7/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736723:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736735:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736741:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736746:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736750:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.736766:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6363171 i_blocks 216 00000020:00010000:0.0:1734016145.736770:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79ca910 (ffff8800a8a54f38/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.736773:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79ca910 (ffff8800a8a54f38/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.736775:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55330 (ffff8800a8a545a8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.736779:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55370 (ffff8800a8a542d8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.736781:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79ca910 (ffff8800a8a54f38/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.736783:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55330 (ffff8800a8a54f38/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.736786:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55370 (ffff8800a8a54f38/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.736796:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.736815:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016145.736819:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.736823:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.736828:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.736831:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55330, osc lock ffff8800a9f23690, flags 0x400001002 00000008:00010000:0.0:1734016145.736833:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.736834:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55370, osc lock ffff8800a9f231d0, flags 0x400001002 00010000:00010000:0.0:1734016145.736838:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736844:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736850:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 7/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736855:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736862:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736866:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736872:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.736877:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.736885:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6363171 i_blocks 216 00000020:00010000:0.0:1734016145.736886:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79ca910 (ffff8800a8a54f38/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.736888:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79ca910 (ffff8800a8a54f38/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.736890:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55330 (ffff8800a8a542d8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.736892:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55370 (ffff8800a8a545a8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.736894:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79ca910 (ffff8800a8a54f38/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.736895:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55330 (ffff8800a8a54f38/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.736897:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55370 (ffff8800a8a54f38/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.736901:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.736905:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 6363171 00000080:00200008:0.0:1734016145.736918:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016145.736923:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016145.736934:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016145.736942:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016145.736970:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016145.736976:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016145.736987:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016145.737007:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334009344:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.737080:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55300 going to sleep for 11 seconds 00000100:00100000:0.0:1734016145.742624:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334009344:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.742645:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.742655:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.742667:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.742675:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.742685:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.742691:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016145.742697:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016145.742706:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78e3b80 x1818247334009344/t0(0) o101->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016161 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016145.742715:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016145.742725:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.742734:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016145.742785:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016145.742790:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e75ec 00000080:00010000:0.0:1734016145.742794:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016145.742796:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e75ec from it ffff8801369fbcd8 00010000:00010000:0.0:1734016145.742799:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.742805:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016145.742833:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.742844:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.742864:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.742868:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55330, osc lock ffff8800a9f231d0, flags 0x400001002 00000008:00010000:0.0:1734016145.742872:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.742874:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55370, osc lock ffff8800a9f23690, flags 0x400001002 00010000:00010000:0.0:1734016145.742894:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.742901:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.743715:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.743723:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.743759:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334009600:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.743822:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016145.743828:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334009472:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016145.749287:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335441152 00000100:00100000:3.0:1734016145.749300:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335441152 00000100:00100000:3.0:1734016145.749346:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12833 00000100:00100000:3.0:1734016145.749353:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335441152:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.749368:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.749377:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.749401:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.749403:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2168867 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000020:00010000:3.0:1734016145.749406:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.749407:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2168867 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000100:00100000:3.0:1734016145.749449:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335441152:12345-192.168.201.116@tcp:106: Request processed in 97us (1639us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.749455:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12833 00000100:00100000:3.0:1734016145.750420:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335441280 00000100:00100000:3.0:1734016145.750424:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335441280 00000100:00100000:3.0:1734016145.750447:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12834 00000100:00100000:3.0:1734016145.750451:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335441280:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.750460:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.750481:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.750494:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.750495:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000020:00010000:3.0:1734016145.750498:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 4, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016145.750499:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000100:00100000:3.0:1734016145.750517:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335441280:12345-192.168.201.116@tcp:106: Request processed in 66us (608us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.750522:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12834 00010000:00010000:0.0:1734016145.752739:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.752753:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.752765:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.752771:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.752777:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.752790:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016145.752797:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.752803:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.752810:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.752816:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.752820:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.752849:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016145.752855:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.752861:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e75f3 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.752888:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334009472:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.753858:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.753867:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.753878:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.753884:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.753890:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.753897:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2168867, setting rss=2168867; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016145.753904:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.753910:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.753916:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.753921:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.753924:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.753929:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016145.753932:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.753941:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1f80/0xa50fd972cb8e75fa lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.753955:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334009600:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016145.758580:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335441408 00000100:00100000:3.0:1734016145.758589:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335441408 00000100:00100000:3.0:1734016145.758898:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335441536 00000100:00100000:3.0:1734016145.758901:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335441536 00000100:00100000:3.0:1734016145.758914:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12835 00000100:00100000:3.0:1734016145.758921:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335441408:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.758934:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.758944:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.758964:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.758965:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000020:00010000:3.0:1734016145.758969:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 4, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016145.758969:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000100:00100000:3.0:1734016145.759005:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335441408:12345-192.168.201.116@tcp:106: Request processed in 85us (790us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.759011:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12835 00000100:00100000:3.0:1734016145.759021:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12836 00000100:00100000:3.0:1734016145.759024:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335441536:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.759029:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.759035:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.759044:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.759045:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2168867 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000020:00010000:3.0:1734016145.759047:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.759048:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2168867 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000100:00100000:3.0:1734016145.759069:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7684:x1818247335441536:12345-192.168.201.116@tcp:106: Request processed in 44us (770us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.759073:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12836 00010000:00010000:3.0:1734016145.762098:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.762113:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.762125:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.762131:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.762136:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.762148:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.762155:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.762160:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.762167:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.762173:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.762178:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.762187:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.762193:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.762198:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367180/0xa50fd972cb8e7601 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.762375:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334009728:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016145.763009:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.763018:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.763025:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.763032:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.763037:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.763045:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2168867, setting rss=2168867; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.763068:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.763074:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.763079:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.763087:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.763090:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.763096:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.763099:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.763104:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7608 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.763123:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334009856:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016145.769965:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 8/2,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.769978:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 7/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.770310:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8801369d4540 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.770360:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3006464 3010559) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 7/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016145.770494:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981f80 - object 0x240000400:6 offset 0<>4194304 00000100:00100000:3.0:1734016145.770511:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016145.770522:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334010112:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016145.846363:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8801369d4540 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.846872:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8801369d4540 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.846899:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334010112:192.168.201.116@tcp:3:fsx.0 00010000:00010000:0.0:1734016145.847090:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.847102:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.847111:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.847117:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.847123:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.849830:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.849851:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.851262:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334010368:192.168.201.116@tcp:3:fsx.0 00000020:00010000:0.0:1734016145.851450:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32128 (ffff8800a8a54518/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.851455:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32128 (ffff8800a8a54518/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.851459:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811f30 (ffff88012dc12758/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016145.851479:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.851489:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016145.851502:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32128 (ffff8800a8a54518/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.851505:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811f30 (ffff8800a8a54518/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.851515:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 4818 bytes 00000080:00200000:0.0:1734016145.851529:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016145.851533:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 4818, restart: 0 00000080:00200000:0.0:1734016145.851538:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 4818 00000080:00200008:0.0:1734016145.851563:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016145.851570:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016145.851588:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016145.851600:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.851608:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016145.851620:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016145.851624:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e75ec 00000080:00010000:0.0:1734016145.851630:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e75ec from it ffff8801369fbcd8 00010000:00010000:0.0:1734016145.851633:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.851639:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e75ec lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc271179db expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016145.851659:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.851669:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.851685:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.851688:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55b70, osc lock ffff8800a9f230a0, flags 0x400001002 00000008:00010000:0.0:1734016145.851694:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.851695:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55bb0, osc lock ffff8800a9f237c0, flags 0x400001002 00010000:00010000:0.0:1734016145.851718:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.851725:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.851770:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016145.851777:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334010496:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.851807:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.851814:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.851867:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334010624:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.870836:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.870857:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.870870:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.870876:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.870882:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.870897:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2168867, setting rss=2168867; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016145.870904:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.870910:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.870915:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.870921:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3a80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.870927:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.870967:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016145.870972:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.870978:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7616 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.871009:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334010624:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.871025:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.871031:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.871038:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.871043:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.871049:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.871054:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016145.871089:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.871096:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.871101:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.871105:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c2d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.871107:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.871111:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016145.871114:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.871118:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e760f lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.871131:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334010496:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.871743:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334010880:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.871792:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016145.871797:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334010752:192.168.201.116@tcp:101:fsx.0 00000008:00010000:0.0:1734016145.880026:0:1982:0:(osc_cache.c:824:osc_extent_finish()) ### no dlm lock: extent: ffff8800a8a550c0 00000008:00010000:0.0:1734016145.880247:0:1982:0:(osc_cache.c:484:__osc_extent_remove()) ### no dlm lock: extent: ffff8800a8a550c0 00000100:00100000:0.0:1734016145.880266:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334010368:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016145.891428:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335442304 00000100:00100000:3.0:1734016145.891432:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335442304 00000100:00100000:3.0:1734016145.891674:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12842 00000100:00100000:3.0:1734016145.891678:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335442304:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.891684:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.891691:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.891703:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1555, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.891705:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000020:00010000:3.0:1734016145.891708:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 4, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016145.891709:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000100:00100000:3.0:1734016145.891725:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335442304:12345-192.168.201.116@tcp:106: Request processed in 47us (464us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.891730:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12842 00010000:00010000:3.0:1734016145.895548:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.895573:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.895587:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.895594:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.895600:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.895613:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.895621:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.895626:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.895633:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.895640:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.895645:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.895657:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.895662:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.895668:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa364fc0/0xa50fd972cb8e762b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.897089:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334011008:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016145.897124:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.897133:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.897142:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.897148:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.897154:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016145.897165:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2168867, setting rss=2168867; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016145.897185:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.897191:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.897197:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.897202:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016145.897205:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.897213:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016145.897216:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.897222:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7632 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.897239:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334011136:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016145.897367:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 6363171 i_blocks 216 00000020:00010000:0.0:1734016145.897374:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79ca480 (ffff8800a8a547e8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016145.897377:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79ca480 (ffff8800a8a547e8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.897383:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7cf0 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.897386:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7d30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016145.897389:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79ca480 (ffff8800a8a547e8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.897391:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7cf0 (ffff8800a8a547e8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016145.897394:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7d30 (ffff8800a8a547e8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016145.897408:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016145.897440:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016145.897445:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016145.897455:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016145.897503:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.897506:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a550f0, osc lock ffff8800a9f23c80, flags 0x400001002 00000008:00010000:0.0:1734016145.897512:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016145.897513:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55130, osc lock ffff8800a9f23db0, flags 0x400001002 00010000:00010000:0.0:1734016145.897542:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e7639 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.897550:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e7639 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.897605:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7640 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.897611:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e7640 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016145.900881:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335442432 00000100:00100000:3.0:1734016145.900893:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335442432 00000100:00100000:3.0:1734016145.900953:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12843 00000100:00100000:3.0:1734016145.900959:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335442432:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.900976:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.900991:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.901021:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1555, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.901023:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000020:00010000:3.0:1734016145.901026:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 4, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016145.901027:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 200 00000100:00100000:3.0:1734016145.901089:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335442432:12345-192.168.201.116@tcp:106: Request processed in 129us (374us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.901096:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12843 00000100:00100000:3.0:1734016145.902709:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335442560 00000100:00100000:3.0:1734016145.902718:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335442560 00000100:00100000:3.0:1734016145.902758:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12844 00000100:00100000:3.0:1734016145.902764:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335442560:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016145.902776:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016145.902786:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e7598 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117964 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016145.902810:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1555, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016145.902812:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2168867 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000020:00010000:3.0:1734016145.902815:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016145.902816:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2168867 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000100:00100000:3.0:1734016145.902854:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335442560:12345-192.168.201.116@tcp:106: Request processed in 90us (417us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.904021:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12844 00000100:00100000:3.0:1734016145.913644:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016145.913665:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335442688 00000100:00100000:3.0:1734016145.913675:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335442688 00000100:00100000:3.0:1734016145.913733:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12845 00000100:00100000:3.0:1734016145.913742:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f85a050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:8c598caf-7141-4c19-8566-0b7731516db9+5:6479:x1818247335442688:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016145.913761:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016145.913836:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016145.913843:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f85a050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:8c598caf-7141-4c19-8566-0b7731516db9+5:6479:x1818247335442688:12345-192.168.201.116@tcp:104: Request processed in 102us (650us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016145.913851:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12845 00010000:00010000:0.0:1734016145.914035:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016145.914050:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.914066:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa10571e0) 00010000:00010000:0.0:1734016145.914071:0:5248:0:(ldlm_request.c:1579:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.914079:0:5248:0:(ldlm_request.c:1589:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.914086:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.914113:0:5248:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.914156:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 2/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.914162:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016145.916340:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.916352:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e75e5 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc271179cd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016145.916385:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.916388:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.916390:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016145.919112:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334011648:192.168.201.116@tcp:36:fsx.0 00000080:00200000:0.0:1734016145.919160:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016145.919166:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016145.919187:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016145.919239:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1734016145.919254:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.919258:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55270, osc lock ffff8800a9f23db0, flags 0x0 00000008:00010000:0.0:1734016145.919261:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016145.919262:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a552b0, osc lock ffff8800a9f23c80, flags 0x0 00010000:00010000:0.0:1734016145.919294:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.919305:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.919337:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.919356:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1880 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334011904:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.919559:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55540 going to sleep for 11 seconds 00000100:00100000:3.0:1734016145.922330:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016145.922768:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016145.922776:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.922783:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1b00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016145.922788:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.922848:0:4750:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.922855:0:4750:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.922867:0:4750:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.922890:0:4750:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800a8a55cc0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.922925:0:4750:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1040384 1044479) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 6/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016145.922998:0:4750:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e3b80 - object 0x240000400:6 offset 1040384<>3846144 00000100:00100000:0.0:1734016145.929378:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1880 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334011904:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.929415:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.929427:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.929435:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016145.935734:0:4750:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016145.939692:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016145.939773:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.939783:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016145.939817:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.939826:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7591 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc2711795d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.939850:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.939852:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016145.939856:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016145.958154:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016145.958696:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.958706:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.958733:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016145.958745:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.958763:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.958769:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.958804:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.958809:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.958843:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016145.958864:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1880 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334012288:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016145.958904:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a550c0 going to sleep for 11 seconds 00000100:00100000:3.0:1734016145.966223:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016145.981846:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1880 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334012288:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016145.981886:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.981899:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016145.981907:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.002247:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016146.002753:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.002764:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.002790:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.002804:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 2168867, setting rss=2168867; kms=2168867, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.002815:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.002822:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.016934:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 240 00000020:00010000:0.0:1734016146.016947:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.016953:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.016957:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55270 (ffff8800a8a54878/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.016969:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.016987:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.017004:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a552b0 (ffff8800a8a54008/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.017008:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017014:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.017022:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.017024:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55270 (ffff8800a8a54ab8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.017031:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a552b0 (ffff8800a8a54ab8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.017049:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.017099:0:10406:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8) current size 8110500, valid attrs 2168, mode 0, uid -1227447776, gid -30720, new size 8110500, atime 1734016145.000000000, mtime 1734016145.905000000, ctime 1734016145.905000000, rc 0 00000080:00a00000:0.0:1734016146.017137:0:10406:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800b6e4e0d8 start=0x7f17cf5ca000 end=0x7f17cf5cc000 vm_flags=0xfb 00000080:00200000:0.0:1734016146.017149:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.017159:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.017171:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.017174:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55270, osc lock ffff8800a9f23c80, flags 0x400001002 00000008:00010000:0.0:1734016146.017179:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.017181:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a552b0, osc lock ffff8800a9f23db0, flags 0x400001002 00010000:00010000:0.0:1734016146.017198:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017215:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017225:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017230:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017240:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017246:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017251:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017256:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.017264:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 240 00000020:00010000:0.0:1734016146.017266:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cab58 (ffff8800a8a54ab8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.017268:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cab58 (ffff8800a8a54ab8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.017270:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55270 (ffff8800a8a54008/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.017273:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a552b0 (ffff8800a8a54878/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.017275:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cab58 (ffff8800a8a54ab8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.017276:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55270 (ffff8800a8a54ab8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.017279:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a552b0 (ffff8800a8a54ab8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.017284:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016146.017343:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 1979 00000080:00200000:0.0:1734016146.017350:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.017359:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.017368:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1979, 1979] 00000008:00010000:0.0:1734016146.017375:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.017378:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999030, osc lock ffff8800a9f23db0, flags 0x0 00010000:00010000:0.0:1734016146.017384:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017391:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3911680 3915775) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017398:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017404:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017409:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017448:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017459:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2732032 2736127) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017527:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 7/1,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017533:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 7/1,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.017538:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.019161:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.019172:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.088258:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.088264:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.088268:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999030 (ffff8800a8a54878/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.088277:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088288:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.088303:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.088305:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999030 (ffff8800a8a54ab8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.088324:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016146.088345:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.088358:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.088372:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1979, 1979] 00000008:00010000:0.0:1734016146.088386:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.088392:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999030, osc lock ffff8800a9f23db0, flags 0x0 00010000:00010000:0.0:1734016146.088412:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088424:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3911680 3915775) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088434:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088441:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088446:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.088716:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800a8a550c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.088737:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134a48, count 1, dirtied 1 00000020:00010000:0.0:1734016146.088748:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.088750:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.088752:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999030 (ffff8800a8a54878/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.088756:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088763:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.088771:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.088772:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999030 (ffff8800a8a54ab8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.088778:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016146.088804:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 1980 00000080:00200000:0.0:1734016146.088810:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.088814:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.088818:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1980, 1980] 00000008:00010000:0.0:1734016146.088822:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.088824:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999030, osc lock ffff8800a9f23db0, flags 0x0 00010000:00010000:0.0:1734016146.088829:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088837:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3915776 3919871) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088844:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088853:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088858:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000001:00010000:0.0:1734016146.088867:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.088873:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.088874:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55db0, osc lock ffff8800a9f23c80, flags 0x400001002 00000008:00010000:0.0:1734016146.088877:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.088878:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55df0, osc lock ffff8800a9f237c0, flags 0x400001002 00010000:00010000:0.0:1734016146.088885:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088891:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088898:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088905:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088913:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088922:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088927:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 8/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.088931:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 7/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.088938:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 240 00000020:00010000:0.0:1734016146.088940:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cab58 (ffff8800a8a54ab8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.088942:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cab58 (ffff8800a8a54ab8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.088943:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55db0 (ffff8800a8a54008/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.088945:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55df0 (ffff8800a8a54878/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.088947:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cab58 (ffff8800a8a54ab8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.088948:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55db0 (ffff8800a8a54ab8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.088950:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55df0 (ffff8800a8a54ab8/1) at cl_lock_fini():85 00000008:00010000:0.0:1734016146.088959:0:10406:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800a8a55d80 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.088965:0:10406:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55d80 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.088972:0:10406:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800a8a550c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.088982:0:10406:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800a8a550c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.088989:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134a48, count 1, dirtied 1 00000020:00010000:0.0:1734016146.088993:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.088994:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.088995:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999030 (ffff8800a8a54878/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.088998:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.089004:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.089010:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32c68 (ffff8800a8a54ab8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.089012:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999030 (ffff8800a8a54ab8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.089018:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016146.089030:0:10406:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8), start 8105984, end 8114175, datasync 1 00000080:00200000:0.0:1734016146.089035:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.089038:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1734016146.089048:0:10406:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a550c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.089069:0:10406:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800a8a550c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.089268:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3911680 3915775) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016146.089327:0:10406:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e2a00 - object 0x280000400:8 offset 3911680<>3916196 00000100:00100000:0.0:1734016146.089373:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016146.089386:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334013056:192.168.201.116@tcp:4:fsx.0 00000008:00010000:0.0:1734016146.099337:0:1982:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a550c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.101953:0:1982:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a550c0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.101997:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334013056:192.168.201.116@tcp:4:fsx.0 00000080:00200000:0.0:1734016146.124693:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 256 00000080:00200000:0.0:1734016146.124712:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.124842:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016146.124850:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016146.124873:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016146.124890:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016146.124941:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016146.124948:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016146.124956:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016146.124979:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2d80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334013440:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016146.125063:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55180 going to sleep for 11 seconds 00000100:00100000:0.0:1734016146.138556:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2d80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334013440:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016146.138585:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.138595:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.138608:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.138617:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.138627:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.138634:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.138639:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016146.138648:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78e2d80 x1818247334013440/t0(0) o101->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016162 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016146.138659:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016146.138672:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016146.138683:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016146.138745:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016146.138751:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e7655 00000080:00010000:0.0:1734016146.138755:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016146.138758:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7655 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016146.138761:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016146.138767:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016146.138800:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.138811:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.138828:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.138832:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551b0, osc lock ffff8800a9f23db0, flags 0x400001002 00000008:00010000:0.0:1734016146.138837:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.138838:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551f0, osc lock ffff8800a9f237c0, flags 0x400001002 00010000:00010000:0.0:1734016146.138854:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.138863:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.138873:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.138878:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.138894:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.138900:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.138905:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.138909:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.138921:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 256 00000020:00010000:0.0:1734016146.138926:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79ca480 (ffff8800a8a547e8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.138929:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79ca480 (ffff8800a8a547e8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.138931:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a551b0 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.138934:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a551f0 (ffff8800a8a54878/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.138936:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79ca480 (ffff8800a8a547e8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.138938:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a551b0 (ffff8800a8a547e8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.138941:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a551f0 (ffff8800a8a547e8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.138948:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.138966:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016146.138969:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.138972:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.138975:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.138977:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551b0, osc lock ffff8800a9f237c0, flags 0x400001002 00000008:00010000:0.0:1734016146.138979:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.138980:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551f0, osc lock ffff8800a9f23db0, flags 0x400001002 00010000:00010000:0.0:1734016146.138984:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.138990:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.138995:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.139000:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.139007:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.139012:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.139017:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.139021:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.139028:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 256 00000020:00010000:0.0:1734016146.139030:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79ca480 (ffff8800a8a547e8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.139032:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79ca480 (ffff8800a8a547e8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.139034:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a551b0 (ffff8800a8a54878/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.139036:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a551f0 (ffff88012dc12758/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.139037:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79ca480 (ffff8800a8a547e8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.139039:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a551b0 (ffff8800a8a547e8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.139040:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a551f0 (ffff8800a8a547e8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.139044:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.139048:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8110500 00000080:00a00000:0.0:1734016146.139122:0:10406:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800b6e4e0d8 start=0x7f17cf5ca000 end=0x7f17cf5cc000 vm_flags=0xf9 00000080:00200000:0.0:1734016146.139134:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.139138:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.139144:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.139145:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551b0, osc lock ffff8800a9f23db0, flags 0x400001002 00000008:00010000:0.0:1734016146.139149:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.139150:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551f0, osc lock ffff8800a9f237c0, flags 0x400001002 00010000:00010000:0.0:1734016146.139172:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.139180:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.139256:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.139263:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.139307:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334013696:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016146.139403:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016146.139410:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334013568:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.146589:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016146.146732:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016146.151401:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.151420:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.151433:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.151439:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.151444:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.151459:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016146.152813:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.152826:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.152834:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.152845:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.152851:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.152862:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016146.152867:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.152873:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e765c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.152925:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334013568:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016146.152950:0:1981:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff88012a571000: committing for last_committed 4294969556 gen 2 00010000:00010000:0.0:1734016146.153003:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.153010:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.153019:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.153024:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.153034:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.153042:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3916196, setting rss=3916196; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016146.153070:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.153075:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.153081:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.153085:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c21c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.153088:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.153094:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016146.153096:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.153102:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e7663 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.153115:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334013696:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.157001:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016146.160150:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016146.160166:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.160189:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.160196:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.160224:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016146.161520:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016146.161581:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 2/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.161589:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016146.161615:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.161621:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e764e lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117a44 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.161646:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.161648:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.161650:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016146.161663:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016146.161687:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334013952:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016146.179371:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016146.186945:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334013952:192.168.201.116@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1734016146.187419:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.187432:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.187902:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:0.0:1734016146.187915:0:1982:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55540 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.187960:0:1982:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (913408 917503) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016146.188029:0:1982:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e0700 - object 0x280000400:8 offset 913408<>3919872 00000100:00100000:0.0:1734016146.188062:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334014208:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016146.236577:0:1982:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a55540 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.239642:0:1982:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55540 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.239690:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334014208:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016146.240855:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7cc0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.240879:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (917504 921599) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016146.240918:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982300 - object 0x280000400:8 offset 917504<>921600 00000020:00010000:0.0:1734016146.252100:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a327b8 (ffff8800a8a547e8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.252104:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a327b8 (ffff8800a8a547e8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.252109:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811630 (ffff8800a8a54878/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.252116:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.252127:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.252142:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a327b8 (ffff8800a8a547e8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.252144:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811630 (ffff8800a8a547e8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.252159:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.252336:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016146.252343:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016146.252364:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016146.252378:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016146.252421:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016146.252429:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016146.252438:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016146.252458:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334014592:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016146.252561:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55540 going to sleep for 11 seconds 00000100:00100000:0.0:1734016146.263997:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334014592:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016146.264027:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.264038:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.264061:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.264071:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.264082:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.264088:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.264094:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016146.264102:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78e0700 x1818247334014592/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016162 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016146.264111:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016146.264123:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016146.264134:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016146.264189:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016146.264194:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e7671 00000080:00010000:0.0:1734016146.264200:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016146.264203:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7671 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016146.264206:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016146.264213:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016146.264248:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.264259:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.264285:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.264288:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55570, osc lock ffff8800a9f23c80, flags 0x400001002 00000008:00010000:0.0:1734016146.264294:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.264295:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a555b0, osc lock ffff8800a9f230a0, flags 0x400001002 00010000:00010000:0.0:1734016146.264318:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.264325:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.271511:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.271540:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.271554:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.271562:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.271602:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016146.271627:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334014720:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.277023:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016146.279772:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.279794:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.279806:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.279812:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.279820:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.279838:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016146.279866:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.279873:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.279880:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.279887:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1b00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.279893:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.279903:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016146.279908:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.279914:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c1b00/0xa50fd972cb8e7678 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.279960:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334014720:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.284018:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016146.286961:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 256 00000020:00010000:0.0:1734016146.286968:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c9488 (ffff8800a8a54d88/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.286971:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c9488 (ffff8800a8a54d88/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.286977:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7cf0 (ffff8800a8a54878/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.286980:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7d30 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.286983:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c9488 (ffff8800a8a54d88/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.286985:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7cf0 (ffff8800a8a54d88/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.286989:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7d30 (ffff8800a8a54d88/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.287000:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.287011:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8110500 00000080:00200008:0.0:1734016146.287080:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 3548042, type: SEEK_SET 00000080:00200008:0.0:1734016146.287086:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 3548042, type: SEEK_SET, rc = 3548042 00000080:00200008:0.0:1734016146.287096:0:10406:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 3548042, count: 920 00000080:00200000:0.0:1734016146.287125:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 866 of 906 to 920 bytes 00000080:00200000:0.0:1734016146.287131:0:10406:0:(file.c:2441:ll_do_tiny_write()) result: 0, original count 920 00000080:00200000:0.0:1734016146.287134:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 3548042, bytes: 920 00000080:00200000:0.0:1734016146.287139:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016146.287145:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016146.287151:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [3548042, 3548962) 0 0 00000080:00200000:0.0:1734016146.287155:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016146.287173:0:10406:0:(file.c:1893:ll_file_io_generic()) Range lock [866, 866] 00020000:00200000:0.0:1734016146.287182:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [3548042, 3548962) 3548962, 920 00020000:00200000:0.0:1734016146.287187:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.287206:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [866, 866] 00000008:00010000:0.0:1734016146.287219:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.287224:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999030, osc lock ffff8800a9f23690, flags 0x0 00010000:00010000:0.0:1734016146.287242:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.287254:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3547136 3551231) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.287262:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.287267:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.287272:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.287280:0:10406:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [3548042, 3548962) 00000080:00200000:0.0:1734016146.287287:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 866 of 906 to 920 bytes 00000100:00100000:0.0:1734016146.287393:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:0.0:1734016146.287413:0:1981:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55b40 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.292503:0:10406:0:(rw26.c:1006:ll_write_end()) pos 3548042, len 920, copied 920 00000080:00200000:0.0:1734016146.292512:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a8b4fca8[3 ffff8800b3a5ce90 2 1 ffff8800a79c94e8] 00000080:00200000:0.0:1734016146.292515:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a64c80 1fffff0000080d 3:0 ffff8800a8b4fca8 866 lru 00000080:00200000:0.0:1734016146.292592:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a8b4fd08 866: 1< 1 - - > 2< 3547136 0 4096 0x0 0x40009 | ffff8800aa965cc8 ffff8800aa37b040 > 3< 0 0 > 4< 0 0 8 121901056 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016146.292595:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a8b4fca8 00000080:00200000:0.0:1734016146.292596:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:0.0:1734016146.292604:0:10406:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 1, from 906, to 1826 00000008:00010000:0.0:1734016146.292628:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800a8a55c00 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.292644:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134a48, count 1, dirtied 1 00000080:00200000:0.0:1734016146.292648:0:10406:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 1 pages 920 bytes, tot: 920 00000080:00200000:0.0:1734016146.292651:0:10406:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 920 00000020:00010000:0.0:1734016146.292669:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32a88 (ffff8800a8a54d88/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.292672:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32a88 (ffff8800a8a54d88/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.292675:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999030 (ffff8800a8a54128/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.292681:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.292689:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.292703:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32a88 (ffff8800a8a54d88/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.292705:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999030 (ffff8800a8a54d88/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.292712:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 920 bytes 00000080:00200000:0.0:1734016146.292717:0:10406:0:(file.c:1930:ll_file_io_generic()) Range unlock [866, 866] 00000080:00200000:0.0:1734016146.292728:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016146.292731:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 920, restart: 0 00000080:00200000:0.0:1734016146.292744:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 256 00000080:00200000:0.0:1734016146.292749:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 920 00000080:00200008:0.0:1734016146.292768:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016146.292774:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016146.292790:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016146.292804:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016146.292813:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016146.292822:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016146.292825:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e7671 00000080:00010000:0.0:1734016146.292829:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7671 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016146.292832:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016146.292836:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016146.292851:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.292862:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.292876:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.292878:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55b70, osc lock ffff8800a9f23690, flags 0x400001002 00000008:00010000:0.0:1734016146.292884:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.292885:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55bb0, osc lock ffff8800a9f231d0, flags 0x400001002 00010000:00010000:0.0:1734016146.292904:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.292914:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.292960:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.292966:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.292974:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.292979:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.292998:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016146.293008:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334015232:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.297624:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016146.300638:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.300656:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.300670:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.300681:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.300686:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.300701:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016146.300725:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.300731:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.300738:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.300745:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c06c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.300749:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.300758:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016146.300764:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.300769:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c06c0/0xa50fd972cb8e7686 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.301636:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334015232:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.306615:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016146.311496:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 256 00000020:00010000:0.0:1734016146.311505:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c9488 (ffff8800a8a54d88/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.311508:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c9488 (ffff8800a8a54d88/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.311513:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee76f0 (ffff8800a8a54878/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.311517:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7730 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.311520:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c9488 (ffff8800a8a54d88/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.311521:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee76f0 (ffff8800a8a54d88/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.311524:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7730 (ffff8800a8a54d88/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.311538:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.311550:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8110500 00000080:00a00000:0.0:1734016146.311631:0:10406:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800b6e4e0d8 start=0x7f17cf5cb000 end=0x7f17cf5cc000 vm_flags=0xf9 00000080:00200000:0.0:1734016146.311642:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.311653:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.311670:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.311673:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55630, osc lock ffff8800a9f238f0, flags 0x400001002 00000008:00010000:0.0:1734016146.311680:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.311682:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55670, osc lock ffff8800a9f23b50, flags 0x400001002 00010000:00010000:0.0:1734016146.311707:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.311714:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.311785:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.311794:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.311802:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.311808:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.311837:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016146.311850:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334015488:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.319641:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016146.325432:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.325451:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.325564:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.325571:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.325577:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.325594:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016146.325853:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.325860:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.325866:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.325872:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c2f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.325878:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.325888:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016146.325893:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.325898:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2f40/0xa50fd972cb8e7694 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.328154:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334015488:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.331611:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016146.332653:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016146.332660:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.332672:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c2ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.332678:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.332723:0:4750:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55c00 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.332734:0:4750:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800a8a55c00 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.332779:0:4750:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3547136 3551231) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016146.332837:0:4750:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e0700 - object 0x240000400:6 offset 3547136<>3551232 00000100:00100000:3.0:1734016146.336743:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334015616:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016146.336781:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [569344->573439] (req 569344->573439) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.336794:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [569344->573439] (req 569344->573439) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.336801:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [569344->573439] (req 569344->573439) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.350658:0:4750:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016146.350739:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016146.350784:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.350791:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016146.350808:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.350815:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2ac0/0xa50fd972cb8e7647 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117a3d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.350847:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.350849:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.350851:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016146.357260:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016146.359780:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.359796:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.359829:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.359844:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.359855:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.359860:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.365433:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32a88 (ffff8800a8a54d88/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.365438:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32a88 (ffff8800a8a54d88/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.365442:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811930 (ffff8800a8a54878/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.365451:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.365498:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.365517:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32a88 (ffff8800a8a54d88/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.365520:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811930 (ffff8800a8a54d88/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.365537:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000020:00010000:0.0:1734016146.381008:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b6320998 (ffff88012dc12c68/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.381012:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6320998 (ffff88012dc12c68/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.381016:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811230 (ffff88012dc12638/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.381024:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.381032:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.381057:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6320998 (ffff88012dc12c68/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.381059:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811230 (ffff88012dc12c68/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.381076:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016146.381127:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 389 00000080:00200000:0.0:1734016146.381135:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.381150:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.381161:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [389, 389] 00000008:00010000:0.0:1734016146.381173:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.381177:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369995b0, osc lock ffff8800a9f23560, flags 0x0 00010000:00010000:0.0:1734016146.381195:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.381208:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1593344 1597439) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.381216:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.381222:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.381227:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.385161:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b6320998 (ffff88012dc12c68/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.385165:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6320998 (ffff88012dc12c68/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.385170:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369995b0 (ffff88012dc12638/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.385177:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385186:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.385200:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6320998 (ffff88012dc12c68/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.385202:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369995b0 (ffff88012dc12c68/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.385217:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.385384:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016146.385391:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016146.385411:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016146.385427:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016146.385436:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016146.385447:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016146.385449:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e7671 00000080:00010000:0.0:1734016146.385455:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7671 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016146.385457:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016146.385462:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2d00/0xa50fd972cb8e7671 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117a75 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016146.385500:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.385511:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.385531:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.385533:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551b0, osc lock ffff8800a9f23560, flags 0x400001002 00000008:00010000:0.0:1734016146.385541:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.385542:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551f0, osc lock ffff8800a9f23a20, flags 0x400001002 00010000:00010000:0.0:1734016146.385556:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385563:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385577:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385583:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385595:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385601:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385607:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385613:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.385626:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 264 00000020:00010000:0.0:1734016146.385628:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed488 (ffff88012dc12c68/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.385630:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed488 (ffff88012dc12c68/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.385631:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a551b0 (ffff88012dc12638/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.385634:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a551f0 (ffff8800a8a54d88/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.385635:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed488 (ffff88012dc12c68/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.385637:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a551b0 (ffff88012dc12c68/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.385641:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a551f0 (ffff88012dc12c68/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.385647:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.385658:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016146.385662:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.385665:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.385668:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.385670:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551b0, osc lock ffff8800a9f23a20, flags 0x400001002 00000008:00010000:0.0:1734016146.385673:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.385674:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551f0, osc lock ffff8800a9f23560, flags 0x400001002 00010000:00010000:0.0:1734016146.385678:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385684:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385690:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385695:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385703:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385709:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385715:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385724:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.385731:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 264 00000020:00010000:0.0:1734016146.385733:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed488 (ffff88012dc12c68/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.385734:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed488 (ffff88012dc12c68/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.385736:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a551b0 (ffff8800a8a54d88/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.385738:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a551f0 (ffff88012dc12638/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.385740:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed488 (ffff88012dc12c68/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.385742:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a551b0 (ffff88012dc12c68/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.385744:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a551f0 (ffff88012dc12c68/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.385747:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.385752:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8110500 00000080:00200000:0.0:1734016146.385811:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016146.385814:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.385823:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.385832:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1086, 1086] 00000008:00010000:0.0:1734016146.385840:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.385843:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369995b0, osc lock ffff8800a9f23560, flags 0x0 00010000:00010000:0.0:1734016146.385849:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800a9dd5400 type 11 mode 2 res 10737419264/8 (253952 258047) 00010000:00010000:0.0:1734016146.385862:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385869:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [253952->258047] (req 253952->258047) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.385899:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [253952->258047] (req 253952->258047) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.385919:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1180 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334017024:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016146.386064:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55180 going to sleep for 11 seconds 00000100:00100000:3.0:1734016146.389807:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016146.389861:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016146.390170:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12862 00000100:00100000:3.0:1734016146.390175:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335444864:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016146.390182:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781f840/0xa50fd972cb8e766a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 913408->917503) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117a67 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.390747:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016146.390753:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335444864:12345-192.168.201.116@tcp:104: Request processed in 577us (891us total) trans 0 rc 0/0 00010000:00010000:0.0:1734016146.391016:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1734016146.391025:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781c6c0/0xa50fd972cb8e76a9 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117aa6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.391032:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781c6c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.391038:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781c6c0/0xa50fd972cb8e76a9 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x5e3a3adc27117aa6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.391065:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016146.391080:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016146.391128:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781c6c0/0xa50fd972cb8e76a9 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117aa6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.391133:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1734016146.391166:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12862 00010000:00010000:0.0:1734016146.394097:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781c6c0/0xa50fd972cb8e76a9 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117aa6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.394109:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781c6c0/0xa50fd972cb8e76a9 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117aa6 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.394149:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.394151:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.394155:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016146.395973:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1180 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334017024:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016146.396014:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [253952->258047] (req 253952->258047) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.396027:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [253952->258047] (req 253952->258047) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.396036:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [253952->258047] (req 253952->258047) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.396241:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334017152:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016146.396353:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016146.396361:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334017280:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016146.401640:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016146.401763:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334017152:192.168.201.116@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1734016146.410672:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.410687:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.410720:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.410733:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 3916196, setting rss=3916196; kms=3916196, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.410751:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.410757:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.410781:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8110500 i_blocks 264 00000100:00100000:0.0:1734016146.410866:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334017408:192.168.201.116@tcp:22:fsx.0 00000100:00100000:0.0:1734016146.417015:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334017408:192.168.201.116@tcp:22:fsx.0 00000100:00100000:0.0:1734016146.417030:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334017280:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016146.418104:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334017536:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.426359:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016146.426383:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335445120 00000100:00100000:3.0:1734016146.426392:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335445120 00000100:00100000:3.0:1734016146.426444:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12864 00000100:00100000:3.0:1734016146.426452:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800ac556450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335445120:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016146.426484:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.426496:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016146.426524:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016146.426526:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000020:00010000:3.0:1734016146.426529:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016146.426530:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000100:00100000:3.0:1734016146.426583:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800ac556450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335445120:12345-192.168.201.116@tcp:106: Request processed in 131us (1572us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.426589:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12864 00010000:00010000:0.0:1734016146.429979:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.430000:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.430015:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.430021:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.430031:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.430082:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3916196, setting rss=3916196; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016146.430106:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.430112:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.430118:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.430125:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781cfc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.430130:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.430140:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016146.430145:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.430150:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781cfc0/0xa50fd972cb8e76b7 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.430187:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334017536:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.440130:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335445248 00000100:00100000:3.0:1734016146.440141:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335445248 00000100:00100000:3.0:1734016146.440194:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12865 00000100:00100000:3.0:1734016146.440202:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335445248:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016146.440222:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.440233:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016146.440262:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016146.440264:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000020:00010000:3.0:1734016146.440267:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016146.440269:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000100:00100000:3.0:1734016146.440332:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335445248:12345-192.168.201.116@tcp:106: Request processed in 130us (1491us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.440338:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12865 00000100:00100000:3.0:1734016146.453650:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335445376 00000100:00100000:3.0:1734016146.453661:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335445376 00000100:00100000:3.0:1734016146.453714:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12866 00000100:00100000:3.0:1734016146.453721:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335445376:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016146.453737:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.453748:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016146.453774:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016146.453777:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000020:00010000:3.0:1734016146.453779:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016146.453781:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000100:00100000:3.0:1734016146.453831:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335445376:12345-192.168.201.116@tcp:106: Request processed in 111us (437us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.453838:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12866 00010000:00010000:3.0:1734016146.456823:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.456846:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.456861:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.456868:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.456873:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016146.456888:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3916196, setting rss=3916196; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016146.456912:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.456918:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.456925:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.456932:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016146.456938:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.456951:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016146.456958:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.456963:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e76c5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.458961:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334017792:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016146.465133:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016146.465458:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.465506:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334018048:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016146.472338:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335445504 00000100:00100000:3.0:1734016146.472348:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335445504 00000100:00100000:3.0:1734016146.472402:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12867 00000100:00100000:3.0:1734016146.472410:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335445504:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016146.472426:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.472438:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016146.472509:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016146.472511:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000020:00010000:3.0:1734016146.472515:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016146.472516:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000100:00100000:3.0:1734016146.472570:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335445504:12345-192.168.201.116@tcp:106: Request processed in 161us (358us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.472577:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12867 00010000:00010000:3.0:1734016146.476973:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.476996:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.477009:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.477015:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.477022:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016146.477039:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3916196, setting rss=3916196; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016146.477072:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.477081:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.477087:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.477093:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016146.477098:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.477108:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016146.477113:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.477118:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e76cc lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.477160:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334018176:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.486297:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335445632 00000100:00100000:3.0:1734016146.486308:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335445632 00000100:00100000:3.0:1734016146.486353:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12868 00000100:00100000:3.0:1734016146.486361:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335445632:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016146.486378:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.486393:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016146.486421:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016146.486423:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000020:00010000:3.0:1734016146.486426:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016146.486427:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000100:00100000:3.0:1734016146.486513:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335445632:12345-192.168.201.116@tcp:106: Request processed in 153us (439us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.486519:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12868 00010000:00010000:3.0:1734016146.498084:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.498106:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.498120:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.498125:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.498131:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016146.498146:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3916196, setting rss=3916196; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016146.498186:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.498192:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.498199:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.498205:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016146.498209:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.498219:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016146.498224:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.498229:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e76d3 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.498273:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334018304:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.505884:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335445760 00000100:00100000:3.0:1734016146.505900:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335445760 00000100:00100000:3.0:1734016146.505960:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12869 00000100:00100000:3.0:1734016146.505967:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335445760:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016146.505986:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.506000:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016146.506029:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016146.506031:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000020:00010000:3.0:1734016146.506034:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016146.506035:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3916196 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 32 00000100:00100000:3.0:1734016146.506096:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335445760:12345-192.168.201.116@tcp:106: Request processed in 129us (1272us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.506102:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12869 00010000:00010000:3.0:1734016146.508548:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.508568:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.508579:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.508585:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.508591:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016146.508607:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3916196, setting rss=3916196; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016146.508628:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.508635:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.508641:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.508648:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016146.508653:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.508664:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016146.508669:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.508675:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e76da lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.508701:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334018432:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.513960:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335445888 00000100:00100000:3.0:1734016146.513972:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335445888 00000100:00100000:3.0:1734016146.514022:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12870 00000100:00100000:3.0:1734016146.514030:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335445888:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016146.514054:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.514134:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016146.514140:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335445888:12345-192.168.201.116@tcp:104: Request processed in 111us (913us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.514147:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12870 00010000:00010000:0.0:1734016146.514299:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016146.514307:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.514315:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.514320:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.514347:0:4750:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016146.514361:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016146.514403:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.514408:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016146.520224:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.520242:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0fc0/0xa50fd972cb8e76b0 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 253952->258047) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117ab4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.520286:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.520288:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.520291:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016146.522260:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016146.553210:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:3.0:1734016146.553239:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e76e1 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1769471) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ae5 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.553358:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1761280 1765375) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e76e1 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1769471) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ae5 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016146.553603:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7980000 - object 0x280000400:8 offset 0<>3919872 00000100:00100000:3.0:1734016146.553624:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016146.553635:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334018944:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016146.659914:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### no dlm lock: extent: ffff8800b6ee7cc0 00000008:00010000:3.0:1734016146.660206:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### no dlm lock: extent: ffff8800b6ee7cc0 00000100:00100000:3.0:1734016146.660235:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334019200:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016146.690825:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e76e1 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1769471) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ae5 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016146.692019:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e76e1 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1769471) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117ae5 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.692068:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334018944:192.168.201.116@tcp:3:fsx.0 00010000:00010000:3.0:1734016146.697550:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e76e8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.697571:0:1984:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e76e8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117aec expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.697587:0:1984:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e76e8 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117aec expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.697597:0:1984:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e76e8 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117aec expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016146.697610:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 3916196, setting rss=3916196; kms=3916196, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e76e8 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117aec expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.697635:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e76e8 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117aec expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.697642:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e76e8 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117aec expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.697648:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e76e8 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117aec expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.697654:0:1984:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e76e8 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117aec expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.697676:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334019328:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016146.707186:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335446144 00000100:00100000:3.0:1734016146.707195:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335446144 00000100:00100000:3.0:1734016146.707238:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12872 00000100:00100000:3.0:1734016146.707245:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:13956:x1818247335446144:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016146.707258:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e7655 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117a52 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016146.707322:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016146.707327:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:13956:x1818247335446144:12345-192.168.201.116@tcp:104: Request processed in 83us (273us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.707333:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12872 00000100:00100000:0.0:1734016146.718383:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334019584:192.168.201.116@tcp:36:fsx.0 00000080:00200000:0.0:1734016146.718425:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016146.718431:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.718453:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.719342:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1734016146.719358:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.719363:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551b0, osc lock ffff8800a9f23a20, flags 0x0 00000008:00010000:0.0:1734016146.719366:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.719367:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551f0, osc lock ffff8800a9f22f70, flags 0x0 00010000:00010000:0.0:1734016146.719382:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800ac554800 type 11 mode 2 res 9663677440/6 (0 18446744073709551615) 00010000:00010000:0.0:1734016146.719394:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.719404:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.719440:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.719458:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2680 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334019840:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016146.719536:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55e40 going to sleep for 11 seconds 00000100:00100000:3.0:1734016146.724993:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016146.725062:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016146.725092:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335446272 00000100:00100000:3.0:1734016146.725101:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335446272 00000100:00100000:3.0:1734016146.725130:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335446400 00000100:00100000:3.0:1734016146.725131:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335446400 00000100:00100000:3.0:1734016146.725144:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12873 00000100:00100000:3.0:1734016146.725150:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335446272:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016146.725168:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.725249:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016146.725255:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335446272:12345-192.168.201.116@tcp:104: Request processed in 105us (271us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.725262:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12873 00000100:00100000:3.0:1734016146.725276:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12874 00000100:00100000:3.0:1734016146.725279:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335446400:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016146.725287:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.725313:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016146.725316:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335446400:12345-192.168.201.116@tcp:104: Request processed in 37us (255us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.725320:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12874 00010000:00010000:3.0:1734016146.725332:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1734016146.725339:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.725345:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781c240 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016146.725352:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016146.725370:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1734016146.725512:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1734016146.725547:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.725552:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1734016146.725568:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.725578:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c240/0xa50fd972cb8e76a2 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117aad expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.725593:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1734016146.725595:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.725598:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.725797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016146.734196:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:3.0:1734016146.734237:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1734016146.738427:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1734016146.738507:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 2/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.738517:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1734016146.738560:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 1/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.738566:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e769b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 569344->573439) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117a9f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.738579:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016146.738580:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016146.738583:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016146.744329:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2680 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334019840:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016146.744355:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.744364:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.744372:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.751539:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.751553:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.751585:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.751597:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.751607:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.751613:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.751630:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800ac555c00 type 11 mode 2 res 10737419264/8 (0 18446744073709551615) 00010000:00010000:0.0:1734016146.751640:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.751646:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.751680:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.751699:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2680 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334020224:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016146.751747:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55e40 going to sleep for 11 seconds 00000100:00100000:3.0:1734016146.760747:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334019968:192.168.201.116@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1734016146.761551:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016146.764738:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1981:1818247334020352:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016146.764761:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016146.764769:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334020480:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016146.765092:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2680 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334020224:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016146.765115:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.765125:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.765131:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.770861:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334020480:192.168.201.116@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1734016146.781704:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.781707:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.781710:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016146.791529:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016146.791543:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335446912 00000100:00100000:3.0:1734016146.791549:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335446912 00000100:00100000:3.0:1734016146.792950:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12878 00000100:00100000:3.0:1734016146.792959:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f859850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335446912:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016146.792974:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.792984:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.792992:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.793004:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.793013:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.793023:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.793068:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c3840) 00010000:00010000:0.0:1734016146.793105:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.793114:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.793138:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.793150:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 3916196, setting rss=3916196; kms=3916196, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.793161:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.793166:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.793259:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334020864:192.168.201.116@tcp:10:fsx.0 00000100:00100000:3.0:1734016146.793393:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f859850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335446912:12345-192.168.201.116@tcp:105: Request processed in 434us (2015us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.793403:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12878 00000100:00100000:0.0:1734016146.799664:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334020864:192.168.201.116@tcp:10:fsx.0 00000008:00010000:3.0:1734016146.807570:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016146.807611:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee79c0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.807640:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334021120:192.168.201.116@tcp:3:fsx.0 00010000:00010000:3.0:1734016146.810445:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 7/1,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.810460:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.810577:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:3.0:1734016146.810596:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8801369d4f00 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.810632:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (5029888 5033983) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016146.810682:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7980000 - object 0x240000400:6 offset 4194304<>5033984 00000100:00100000:3.0:1734016146.810706:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334021632:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016146.830450:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8801369d4f00 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016146.831418:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8801369d4f00 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.831443:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334021632:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016146.831633:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### no dlm lock: extent: ffff8800b6ee7a80 00000008:00010000:3.0:1734016146.831658:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### no dlm lock: extent: ffff8800b6ee7a80 00000100:00100000:3.0:1734016146.831665:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334021376:192.168.201.116@tcp:3:fsx.0 00000020:00010000:0.0:1734016146.831756:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b63205d8 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.831760:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63205d8 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.831764:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811430 (ffff88012dc12368/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.831771:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831783:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.831803:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63205d8 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.831805:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811430 (ffff88012dc12638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.831820:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016146.831832:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.831845:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.831857:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2252, 2252] 00000008:00010000:0.0:1734016146.831871:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.831875:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999030, osc lock ffff8800a9f22ab0, flags 0x0 00010000:00010000:0.0:1734016146.831888:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831898:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (5029888 5033983) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831906:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831911:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831916:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000001:00010000:0.0:1734016146.831931:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.831935:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.831936:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55e70, osc lock ffff8800a9f224c0, flags 0x400001002 00000008:00010000:0.0:1734016146.831942:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.831943:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55eb0, osc lock ffff8800a9f23300, flags 0x400001002 00010000:00010000:0.0:1734016146.831948:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831955:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831961:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 8/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831967:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 7/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831976:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831983:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831989:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.831999:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.832010:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9227759 i_blocks 264 00000020:00010000:0.0:1734016146.832013:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.832015:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.832017:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55e70 (ffff8800a8a54128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.832019:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55eb0 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.832021:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.832023:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55e70 (ffff88012dc12638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.832025:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55eb0 (ffff88012dc12638/1) at cl_lock_fini():85 00000008:00010000:0.0:1734016146.832062:0:10406:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800a8a55e40 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.832069:0:10406:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55e40 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.832075:0:10406:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.832083:0:10406:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.832100:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134248, count 1, dirtied 1 00000020:00010000:0.0:1734016146.832112:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b63205d8 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.832114:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63205d8 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.832117:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999030 (ffff8800a8a54128/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016146.832120:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.832126:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016146.832133:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63205d8 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.832134:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999030 (ffff88012dc12638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.832141:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016146.832163:0:10406:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8), start 9220096, end 9228287, datasync 1 00000080:00200000:0.0:1734016146.832169:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.832172:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:0.0:1734016146.832184:0:10406:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016146.832199:0:10406:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800b6ee7900 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.834458:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (5025792 5029887) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.848916:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334021888:192.168.201.116@tcp:44:fsx.0 00000080:00200000:0.0:1734016146.848948:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.848964:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.857155:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9227759 i_blocks 280 00000080:00200000:0.0:1734016146.857168:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.857286:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016146.857293:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016146.857308:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016146.857324:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016146.857362:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016146.857369:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016146.857377:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016146.857392:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0a80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334022144:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016146.858031:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55e40 going to sleep for 11 seconds 00000100:00100000:0.0:1734016146.864369:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0a80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334022144:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016146.864391:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.864401:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.864413:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.864420:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.864427:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.864434:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016146.864438:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016146.864446:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78e0a80 x1818247334022144/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016162 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016146.864454:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016146.864462:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016146.864499:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016146.864545:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016146.864549:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e76fd 00000080:00010000:0.0:1734016146.864552:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016146.864555:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e76fd from it ffff8801369fbcd8 00010000:00010000:0.0:1734016146.864557:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016146.864563:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e76fd lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117b0f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016146.864587:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.864597:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.864618:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.864621:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55e70, osc lock ffff8800a9f22ab0, flags 0x400001002 00000008:00010000:0.0:1734016146.864644:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.864645:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55eb0, osc lock ffff8800a9f23300, flags 0x400001002 00010000:00010000:0.0:1734016146.864654:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864663:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864672:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864678:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864690:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864695:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864700:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864705:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.864719:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9227759 i_blocks 280 00000020:00010000:0.0:1734016146.864723:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.864726:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.864727:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55e70 (ffff8800a8a54bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.864729:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55eb0 (ffff8800a8a54638/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.864732:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.864733:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55e70 (ffff8800a8a54128/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.864737:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55eb0 (ffff8800a8a54128/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.864745:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.864760:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016146.864763:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.864766:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.864770:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.864771:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55e70, osc lock ffff8800a9f23300, flags 0x400001002 00000008:00010000:0.0:1734016146.864773:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.864775:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55eb0, osc lock ffff8800a9f22ab0, flags 0x400001002 00010000:00010000:0.0:1734016146.864779:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864784:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864791:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864796:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864803:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864809:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864814:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.864819:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.864826:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9227759 i_blocks 280 00000020:00010000:0.0:1734016146.864828:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.864830:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.864831:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55e70 (ffff8800a8a54638/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.864833:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55eb0 (ffff8800a8a54bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.864835:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c96d0 (ffff8800a8a54128/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.864836:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55e70 (ffff8800a8a54128/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.864838:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55eb0 (ffff8800a8a54128/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.864842:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.864847:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9227759 00000080:00200008:0.0:1734016146.864893:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 489721, type: SEEK_SET 00000080:00200008:0.0:1734016146.864898:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 489721, type: SEEK_SET, rc = 489721 00000080:00200008:0.0:1734016146.864906:0:10406:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 489721, count: 4889 00000080:00200000:0.0:1734016146.864909:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016146.864917:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016146.864940:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 119 00000080:00200000:0.0:1734016146.864948:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 489721, bytes: 4889 00000080:00200000:0.0:1734016146.864951:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016146.864956:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016146.864958:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [489721, 494610) 0 0 00000080:00200000:0.0:1734016146.864960:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.864971:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [489721, 494610) 494610, 4889 00020000:00200000:0.0:1734016146.864976:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.864988:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [119, 120] 00000008:00010000:0.0:1734016146.864994:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.864996:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999030, osc lock ffff8800a9f22ab0, flags 0x0 00010000:00010000:0.0:1734016146.865012:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.865019:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [487424->495615] (req 487424->495615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.865054:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [487424->495615] (req 487424->495615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.865071:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0a80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334022272:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016146.865131:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a550c0 going to sleep for 11 seconds 00000100:00100000:3.0:1734016146.869158:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335447040 00000100:00100000:3.0:1734016146.869168:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335447040 00000100:00100000:3.0:1734016146.869238:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12879 00000100:00100000:3.0:1734016146.869245:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335447040:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016146.869260:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.870347:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016146.870354:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335447040:12345-192.168.201.116@tcp:104: Request processed in 1109us (1390us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.870362:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12879 00010000:00010000:0.0:1734016146.870456:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016146.870476:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.870485:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0000 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.870491:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.870536:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016146.871444:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016146.871508:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 2/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.871515:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016146.871542:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.871548:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0000/0xa50fd972cb8e76ef lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117afa expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.871568:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.871570:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.871572:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016146.876107:0:10406:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0000-osc-ffff88012a5ff800: committing for last_committed 4294968107 gen 2 00000100:00100000:0.0:1734016146.876138:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0a80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334022272:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016146.876160:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [487424->495615] (req 487424->495615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.876169:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [487424->495615] (req 487424->495615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.876176:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [487424->495615] (req 487424->495615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016146.879181:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335447168 00000100:00100000:3.0:1734016146.879188:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335447168 00000100:00100000:3.0:1734016146.879218:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12880 00000100:00100000:3.0:1734016146.879224:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335447168:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016146.879238:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [487424->495615] (req 487424->495615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.879252:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [487424->495615] (req 487424->495615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.879259:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.879267:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.879274:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.879281:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016146.880688:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c3600) 00000100:00100000:3.0:1734016146.880752:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335447168:12345-192.168.201.116@tcp:105: Request processed in 1525us (2173us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016146.880760:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12880 00000100:00100000:0.0:1734016146.887165:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334022400:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016146.973509:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334022784:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016146.986902:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9227759 i_blocks 280 00000020:00010000:0.0:1734016146.986908:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.986911:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.986916:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7630 (ffff8800a8a54bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.986920:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7670 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.986923:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.986925:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7630 (ffff88012dc12638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.986929:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7670 (ffff88012dc12638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.986943:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.986962:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016146.986967:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016146.986976:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016146.986993:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.986995:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a550f0, osc lock ffff8800a9f23300, flags 0x400001002 00000008:00010000:0.0:1734016146.987001:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016146.987002:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55130, osc lock ffff8800a9f224c0, flags 0x400001002 00010000:00010000:0.0:1734016146.987016:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.987025:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.987032:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.987052:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.987064:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.987070:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.987076:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.987081:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.987090:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9227759 i_blocks 280 00000020:00010000:0.0:1734016146.987093:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016146.987094:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.987096:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a550f0 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.987098:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55130 (ffff8800a8a54bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016146.987100:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.987101:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a550f0 (ffff88012dc12638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016146.987104:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55130 (ffff88012dc12638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016146.987109:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016146.987113:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9227759 00000080:00200008:0.0:1734016146.987153:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 8210157, type: SEEK_SET 00000080:00200008:0.0:1734016146.987159:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 8210157, type: SEEK_SET, rc = 8210157 00000080:00200008:0.0:1734016146.987165:0:10406:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 8210157, count: 7160 00000080:00200000:0.0:1734016146.987168:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016146.987174:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016146.987195:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 2004 00000080:00200000:0.0:1734016146.987201:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 8210157, bytes: 7160 00000080:00200000:0.0:1734016146.987204:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016146.987208:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5ce10[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016146.987210:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [8210157, 8217317) 0 0 00000080:00200000:0.0:1734016146.987212:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016146.987221:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [8210157, 8217317) 8217317, 7160 00020000:00200000:0.0:1734016146.987226:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016146.987235:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [2004, 2006] 00000008:00010000:0.0:1734016146.987240:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016146.987242:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369995b0, osc lock ffff8800a9f224c0, flags 0x0 00010000:00010000:0.0:1734016146.987256:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.987264:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [4014080->4026367] (req 4014080->4026367) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.987284:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [4014080->4026367] (req 4014080->4026367) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016146.987299:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334022912:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016146.987336:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55240 going to sleep for 11 seconds 00000100:00100000:3.0:1734016146.990255:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016146.990694:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016146.990701:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.990712:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016146.990717:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016146.990736:0:4750:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016146.991067:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016146.991105:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.991111:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016146.991143:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.991150:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3840/0xa50fd972cb8e76f6 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117b01 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016146.991167:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.991168:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016146.991170:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016146.991182:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016146.991194:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334023040:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016147.007372:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334022912:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016147.007403:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [4014080->4026367] (req 4014080->4026367) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.007416:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [4014080->4026367] (req 4014080->4026367) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.007423:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [4014080->4026367] (req 4014080->4026367) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016147.011981:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335447424 00000100:00100000:3.0:1734016147.011990:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335447424 00000100:00100000:3.0:1734016147.012054:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12882 00000100:00100000:3.0:1734016147.012061:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:9876:x1818247335447424:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016147.012075:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [4014080->4026367] (req 4014080->4026367) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.012087:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [4014080->4026367] (req 4014080->4026367) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.012095:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.012105:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.012112:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.012121:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.012148:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c0240) 00000100:00100000:3.0:1734016147.012195:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:9876:x1818247335447424:12345-192.168.201.116@tcp:105: Request processed in 133us (759us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.012201:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12882 00010000:00010000:0.0:1734016147.012614:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.012622:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.012646:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016147.012657:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.012669:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.012675:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016147.012689:0:10406:0:(vvp_io.c:840:vvp_io_read_start()) f16b.sanityn: read [8210157, 8217317) 00000080:00200000:0.0:1734016147.012715:0:10406:0:(vvp_io.c:899:vvp_io_read_start()) count:7160 iocb pos:8210157 00010000:00010000:0.0:1734016147.012772:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.012784:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.013233:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.013241:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.013259:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.013265:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.125734:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.125755:0:1984:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.125778:0:1984:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.125788:0:1984:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016147.125801:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.125828:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.125835:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.125841:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.125848:0:1984:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016147.125867:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334023680:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016147.143121:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.143138:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.143169:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.143179:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016147.163997:0:1982:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a7b190c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.164060:0:1982:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3006464 3010559) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016147.164272:0:1982:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e1c00 - object 0x280000400:8 offset 0<>4194304 00000100:00100000:0.0:1734016147.164304:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334024064:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016147.165414:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### no dlm lock: extent: ffff8800b6ee7cc0 00000008:00010000:3.0:1734016147.165508:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### no dlm lock: extent: ffff8800b6ee7cc0 00000100:00100000:3.0:1734016147.165522:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334023552:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016147.323889:0:1982:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a7b190c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016147.324243:0:1982:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a7b190c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016147.324272:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334024064:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016147.326400:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8800b6ee76c0 00010000:00010000:3.0:1734016147.326444:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4194304 4198399) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016147.326525:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981c00 - object 0x240000400:6 offset 4194304<>5033984 00000100:00100000:3.0:1734016147.327934:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016147.327941:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334024448:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016147.341589:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016147.341629:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016147.341661:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016147.341676:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335447552 00000100:00100000:3.0:1734016147.341685:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335447552 00000100:00100000:3.0:1734016147.341719:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335447680 00000100:00100000:3.0:1734016147.341721:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335447680 00000100:00100000:3.0:1734016147.341734:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12883 00000100:00100000:3.0:1734016147.341741:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335447552:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016147.341758:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.341810:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016147.341816:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335447552:12345-192.168.201.116@tcp:104: Request processed in 76us (241us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.341822:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12883 00000100:00100000:3.0:1734016147.341837:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12884 00000100:00100000:3.0:1734016147.341840:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335447680:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016147.341847:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.341887:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016147.341892:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335447680:12345-192.168.201.116@tcp:104: Request processed in 50us (263us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.341896:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12884 00010000:00010000:0.0:1734016147.347736:0:4749:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016147.347752:0:4749:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.347766:0:4749:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3600 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016147.347774:0:4749:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7704 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 487424->495615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117b16 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016147.347793:0:4749:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016147.369226:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016147.369301:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 2/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.369313:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016147.369344:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 1/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.369352:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e770b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117b1d expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016147.369375:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016147.369394:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334024832:192.168.201.116@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1734016147.369538:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016147.369540:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016147.369545:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016147.375770:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335447808 00000100:00100000:3.0:1734016147.375784:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335447808 00000100:00100000:3.0:1734016147.375835:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12885 00000100:00100000:3.0:1734016147.375845:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6466:x1818247335447808:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016147.375862:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3678208->3686399] (req 3678208->3686399) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.375875:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3678208->3686399] (req 3678208->3686399) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.375884:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.375894:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.375903:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.375912:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.375939:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366f40) 00000100:00100000:3.0:1734016147.375995:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6466:x1818247335447808:12345-192.168.201.116@tcp:105: Request processed in 150us (368us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.376002:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12885 00010000:00010000:0.0:1734016147.376264:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.376274:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.376298:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016147.376311:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 5033455, setting rss=5033455; kms=5033455, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.376326:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.376332:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016147.376359:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9227759 i_blocks 280 00000100:00100000:0.0:1734016147.376435:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334024960:192.168.201.116@tcp:22:fsx.0 00000100:00100000:0.0:1734016147.385851:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334024960:192.168.201.116@tcp:22:fsx.0 00000100:00100000:0.0:1734016147.385869:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334024832:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016147.390969:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334025216:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016147.396388:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016147.407508:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9227759 i_blocks 296 00000020:00010000:0.0:1734016147.407516:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016147.407520:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.407525:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee76f0 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.407530:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7730 (ffff8800a8a54bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.407533:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.407535:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee76f0 (ffff88012dc12638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.407539:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7730 (ffff88012dc12638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016147.407555:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016147.407578:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016147.407583:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016147.407593:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016147.407611:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016147.407615:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55270, osc lock ffff8800a9f23300, flags 0x400001002 00000008:00010000:0.0:1734016147.407620:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016147.407622:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a552b0, osc lock ffff8800a9f22d10, flags 0x400001002 00010000:00010000:0.0:1734016147.407644:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.407653:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.411380:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.411402:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.411416:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.411424:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016147.411459:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016147.411506:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334025344:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016147.427321:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016147.436143:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.436164:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.436183:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.436189:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.436194:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016147.436208:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5033455, setting rss=5033455; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016147.436233:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.436238:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.436245:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.436250:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3600 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016147.436255:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.436266:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016147.436271:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.436281:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7735 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016147.438278:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334025344:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016147.449340:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016147.457904:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9227759 i_blocks 296 00000020:00010000:0.0:1734016147.457912:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016147.457915:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.457920:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee77b0 (ffff88012dc12368/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.457924:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee77f0 (ffff8800a8a54bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.457927:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.457928:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee77b0 (ffff88012dc12638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.457932:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee77f0 (ffff88012dc12638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016147.457944:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016147.457966:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016147.457971:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016147.457979:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016147.457992:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016147.457995:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f30, osc lock ffff8800a9f23430, flags 0x400001002 00000008:00010000:0.0:1734016147.458001:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016147.458002:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f70, osc lock ffff8800a9f22000, flags 0x400001002 00010000:00010000:0.0:1734016147.458023:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.458039:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.458109:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.458118:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.458126:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.458133:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016147.458161:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016147.458174:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334025600:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016147.466529:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016147.475327:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.475349:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.475367:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.475372:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.475383:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016147.475399:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5033455, setting rss=5033455; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016147.476416:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.476425:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.476431:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.476438:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3600 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016147.476442:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.476453:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016147.476459:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.476477:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3600/0xa50fd972cb8e7743 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016147.480102:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334025600:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016147.490598:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016147.490890:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016147.490897:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7727 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117b40 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016147.490904:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa10571e0) 00010000:00010000:0.0:1734016147.490908:0:4750:0:(ldlm_request.c:1579:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7727 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117b40 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016147.490914:0:4750:0:(ldlm_request.c:1589:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7727 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117b40 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016147.490920:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7727 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x5e3a3adc27117b40 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016147.490943:0:4750:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781c900/0xa50fd972cb8e5f33 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115fb0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016147.490983:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7727 lrc: 2/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117b40 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016147.490989:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016147.491022:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7727 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117b40 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016147.491036:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa3646c0/0xa50fd972cb8e7727 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117b40 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016147.491060:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016147.491061:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016147.491063:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016147.525424:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016147.525443:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335448576 00000100:00100000:3.0:1734016147.525453:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335448576 00000100:00100000:3.0:1734016147.525537:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12891 00000100:00100000:3.0:1734016147.525543:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2ddc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335448576:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016147.525558:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7719 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117b2b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.525631:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016147.525637:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2ddc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335448576:12345-192.168.201.116@tcp:104: Request processed in 94us (1010us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.525643:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12891 00000100:00100000:0.0:1734016147.527591:0:4739:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335448704 00000100:00100000:0.0:1734016147.527601:0:4739:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335448704 00000100:00100000:0.0:1734016147.527674:0:4739:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 231 00000100:00100000:0.0:1734016147.527681:0:4739:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:7683:x1818247335448704:12345-192.168.201.116@tcp:104: 00010000:00010000:0.0:1734016147.527695:0:4739:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.527749:0:4739:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1734016147.527754:0:4739:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:7683:x1818247335448704:12345-192.168.201.116@tcp:104: Request processed in 75us (2788us total) trans 0 rc 0/0 00000100:00100000:0.0:1734016147.527759:0:4739:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 231 00010000:00010000:0.0:1734016147.527801:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016147.527805:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.527811:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0240 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016147.527814:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0240/0xa50fd972cb8e7712 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4014080->4026367) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117b24 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016147.527826:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000100:00100000:3.0:1734016147.554915:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334026368:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016147.562682:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335448832 00000100:00100000:3.0:1734016147.562695:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335448832 00000100:00100000:3.0:1734016147.562754:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12892 00000100:00100000:3.0:1734016147.562761:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6466:x1818247335448832:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016147.562776:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.562786:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.562794:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.562804:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.562812:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.562821:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.562850:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa364900) 00010000:00010000:0.0:1734016147.562877:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.562889:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016147.562897:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6466:x1818247335448832:12345-192.168.201.116@tcp:105: Request processed in 136us (475us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.562902:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12892 00010000:00010000:0.0:1734016147.562918:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016147.562935:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.562952:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.562958:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016147.572347:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334026368:192.168.201.116@tcp:103:ldlm_bl.0 00000080:00200000:0.0:1734016147.589256:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5996913 i_blocks 264 00000020:00010000:0.0:1734016147.589265:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b63205d8 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016147.589269:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63205d8 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.589273:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7930 (ffff88012dc12368/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016147.589282:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.589293:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016147.589308:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7970 (ffff8800a8a54bd8/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016147.589312:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.589318:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016147.589331:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63205d8 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.589333:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7930 (ffff88012dc12638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.589337:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7970 (ffff88012dc12638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016147.589353:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016147.589365:0:10406:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8) current size 5996913, valid attrs 2168, mode 8801, uid -1227447776, gid -30720, new size 5996913, atime 1734016147.000000000, mtime 1734016147.481000000, ctime 1734016147.481000000, rc 0 00000080:00200008:0.0:1734016147.589392:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016147.589398:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016147.589412:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016147.589424:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016147.589456:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016147.589501:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016147.589511:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016147.589531:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334026752:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016147.589588:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1734016147.604696:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335448960 00000100:00100000:3.0:1734016147.604707:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335448960 00000100:00100000:3.0:1734016147.604734:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335449088 00000100:00100000:3.0:1734016147.604736:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335449088 00000100:00100000:3.0:1734016147.604746:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12893 00000100:00100000:3.0:1734016147.604752:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335448960:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016147.604765:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.604774:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016147.604797:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016147.604799:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 1802609 mtime: 1734016147 atime: 1734016146 ctime: 1734016147 blocks: 16 00000020:00010000:3.0:1734016147.604802:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016147.604803:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 1802609 mtime: 1734016147 atime: 1734016146 ctime: 1734016147 blocks: 16 00000100:00100000:3.0:1734016147.604841:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335448960:12345-192.168.201.116@tcp:106: Request processed in 89us (1115us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.604847:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12893 00000100:00100000:3.0:1734016147.604857:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12894 00000100:00100000:3.0:1734016147.604860:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335449088:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016147.604866:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.604872:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016147.604880:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016147.604882:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016146 ctime: 1734016147 blocks: 248 00000020:00010000:3.0:1734016147.604885:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016147.604886:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016146 ctime: 1734016147 blocks: 248 00000100:00100000:3.0:1734016147.604896:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335449088:12345-192.168.201.116@tcp:106: Request processed in 35us (437us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.604900:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12894 00000080:00200000:0.0:1734016147.615494:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5996913 i_blocks 264 00000020:00010000:0.0:1734016147.615500:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c9b60 (ffff8800a8a54638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016147.615503:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c9b60 (ffff8800a8a54638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.615545:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7b19e70 (ffff8800a8a54cf8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.615549:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7b19eb0 (ffff88012dc12518/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.615552:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c9b60 (ffff8800a8a54638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.615553:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7b19e70 (ffff8800a8a54638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.615557:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7b19eb0 (ffff8800a8a54638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016147.615569:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016147.615588:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016147.615592:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016147.615600:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016147.615615:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016147.615618:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a550f0, osc lock ffff8800a9f22850, flags 0x400001002 00000008:00010000:0.0:1734016147.615623:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016147.615624:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55130, osc lock ffff8800a9f22be0, flags 0x400001002 00010000:00010000:0.0:1734016147.615641:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7766 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.615648:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e7766 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.615688:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e776d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.615693:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0480/0xa50fd972cb8e776d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016147.633722:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5996913 i_blocks 264 00000020:00010000:0.0:1734016147.633728:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c9b60 (ffff8800a8a54638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016147.633730:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c9b60 (ffff8800a8a54638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.633734:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a550f0 (ffff88012dc12518/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.633738:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55130 (ffff8800a8a54cf8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.633739:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c9b60 (ffff8800a8a54638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.633741:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a550f0 (ffff8800a8a54638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.633744:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55130 (ffff8800a8a54638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016147.633757:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016147.633768:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 5996913 00000080:00200008:0.0:1734016147.633826:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 97811, type: SEEK_SET 00000080:00200008:0.0:1734016147.633829:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 97811, type: SEEK_SET, rc = 97811 00000080:00200008:0.0:1734016147.633837:0:10406:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 97811, count: 7418 00000080:00200000:0.0:1734016147.633842:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016147.633850:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016147.633878:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 23 00000080:00200000:0.0:1734016147.633887:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 97811, bytes: 7418 00000080:00200000:0.0:1734016147.633890:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016147.633894:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016147.633896:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [97811, 105229) 0 0 00000080:00200000:0.0:1734016147.633898:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016147.633909:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [97811, 105229) 105229, 7418 00020000:00200000:0.0:1734016147.633915:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016147.633929:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [23, 25] 00000008:00010000:0.0:1734016147.633945:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016147.633949:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369990b0, osc lock ffff8800a9f22be0, flags 0x0 00010000:00010000:0.0:1734016147.633966:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.633973:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [94208->106495] (req 94208->106495) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.633997:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [94208->106495] (req 94208->106495) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016147.634012:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2680 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334027392:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016147.634067:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55e40 going to sleep for 11 seconds 00000100:00100000:3.0:1734016147.638808:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335449472 00000100:00100000:3.0:1734016147.638815:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335449472 00000100:00100000:3.0:1734016147.638853:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12897 00000100:00100000:3.0:1734016147.638858:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335449472:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016147.638871:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.638921:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016147.638925:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335449472:12345-192.168.201.116@tcp:104: Request processed in 67us (204us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.638930:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12897 00010000:00010000:0.0:1734016147.639160:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016147.639168:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.639175:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016147.639181:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016147.639203:0:4750:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016147.639216:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016147.639246:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.639251:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016147.641445:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.641453:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e7720 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3678208->3686399) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117b32 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.641647:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016147.641649:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016147.641652:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016147.647936:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2680 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334027392:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016147.647959:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [94208->106495] (req 94208->106495) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.647972:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [94208->106495] (req 94208->106495) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.647978:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [94208->106495] (req 94208->106495) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016147.648041:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016147.648053:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335449600 00000100:00100000:3.0:1734016147.648058:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335449600 00000100:00100000:3.0:1734016147.648086:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12898 00000100:00100000:3.0:1734016147.648091:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6466:x1818247335449600:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016147.648098:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [94208->106495] (req 94208->106495) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.648105:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [94208->106495] (req 94208->106495) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.648115:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.648122:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.648128:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.648134:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.648146:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c0000) 00000100:00100000:3.0:1734016147.648177:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dd450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6466:x1818247335449600:12345-192.168.201.116@tcp:105: Request processed in 86us (331us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.648183:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12898 00010000:00010000:0.0:1734016147.648236:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.648242:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.648258:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016147.648267:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.648277:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.648283:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016147.648293:0:10406:0:(vvp_io.c:840:vvp_io_read_start()) f16b.sanityn: read [97811, 105229) 00000080:00200000:0.0:1734016147.648311:0:10406:0:(vvp_io.c:899:vvp_io_read_start()) count:7418 iocb pos:97811 00010000:00010000:0.0:1734016147.648343:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.648352:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.653312:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.653325:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016147.767434:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a32a88 (ffff8800a8a54638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016147.767438:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a32a88 (ffff8800a8a54638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.767442:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369990b0 (ffff8800a8a54cf8/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016147.767448:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.767456:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016147.767496:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a32a88 (ffff8800a8a54638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.767499:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369990b0 (ffff8800a8a54638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016147.767505:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 7418 bytes 00000080:00200000:0.0:1734016147.767514:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016147.767517:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 7418, restart: 0 00000080:00200000:0.0:1734016147.767522:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 7418 00000080:00200008:0.0:1734016147.767541:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016147.767547:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016147.767561:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016147.767575:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016147.767583:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016147.767593:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016147.767595:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e7751 00000080:00010000:0.0:1734016147.767600:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7751 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016147.767602:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016147.767607:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016147.767622:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016147.767633:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016147.767649:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016147.767652:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a559f0, osc lock ffff8800a9f22be0, flags 0x400001002 00000008:00010000:0.0:1734016147.767657:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016147.767658:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55a30, osc lock ffff8800a9f22850, flags 0x400001002 00010000:00010000:0.0:1734016147.767666:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.767672:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.767684:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.767690:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.767709:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.767714:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016147.767765:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016147.767773:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334027904:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016147.774847:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016147.785265:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.785282:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.785294:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.785300:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.785310:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016147.785323:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 1802609, setting rss=1802609; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016147.785346:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.785352:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.785358:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.785364:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016147.785369:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.785379:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016147.785383:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.785388:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3840/0xa50fd972cb8e777b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016147.785421:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334027904:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016147.802756:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016147.820380:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5996913 i_blocks 264 00000020:00010000:0.0:1734016147.820390:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79cbd98 (ffff8800a8a54cf8/0) at cl_lock_release():233 00000020:00010000:0.0:1734016147.820392:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79cbd98 (ffff8800a8a54cf8/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.820398:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee79f0 (ffff88012dc12098/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.820402:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7a30 (ffff88012dc12248/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.820404:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79cbd98 (ffff8800a8a54cf8/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.820406:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee79f0 (ffff8800a8a54cf8/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.820410:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7a30 (ffff8800a8a54cf8/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016147.820427:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016147.820442:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 5996913 00000100:00100000:3.0:1734016147.829734:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016147.845436:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016147.848420:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016147.848428:0:5248:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.848437:0:5248:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0000 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016147.848443:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016147.848460:0:5248:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016147.852941:0:5248:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016147.852998:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.853008:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016147.870385:0:5248:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.870404:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c0000/0xa50fd972cb8e7774 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 94208->106495) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117b9b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016147.881551:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016147.882082:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.882091:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.882111:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016147.882121:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.882131:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.882138:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016147.882160:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5996913 i_blocks 264 00000020:00010000:0.0:1734016147.905100:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800b6320038 (ffff88012dc12008/0) at cl_lock_release():233 00000020:00010000:0.0:1734016147.905108:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6320038 (ffff88012dc12008/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016147.905112:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811430 (ffff88012dc12998/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016147.905123:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016147.905135:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016147.905153:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6320038 (ffff88012dc12008/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016147.905156:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811430 (ffff88012dc12008/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016147.905179:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016147.905233:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016147.905239:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016147.905259:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016147.905271:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:0.0:1734016147.905309:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c0480/0xa50fd972cb8e7797 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016147.905315:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c0480/0xa50fd972cb8e7797 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00010000:00010000:0.0:1734016147.905324:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c0480/0xa50fd972cb8e7797 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10406 initiator: MDT0 00000100:00100000:0.0:1734016147.905348:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0380 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334028672:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016147.906331:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55a80 going to sleep for 11 seconds 00000100:00100000:3.0:1734016147.925582:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335451264 00000100:00100000:3.0:1734016147.925592:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335451264 00000100:00100000:3.0:1734016147.925633:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12904 00000100:00100000:3.0:1734016147.925640:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335451264:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016147.925656:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.925664:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016147.925686:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016147.925687:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5556270 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 264 00000020:00010000:3.0:1734016147.925690:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016147.925691:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5556270 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 264 00000100:00100000:3.0:1734016147.925734:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335451264:12345-192.168.201.116@tcp:106: Request processed in 93us (306us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.925740:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12904 00000100:00100000:3.0:1734016147.943738:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016147.943752:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335451392 00000100:00100000:3.0:1734016147.943759:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335451392 00000100:00100000:3.0:1734016147.943800:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12905 00000100:00100000:3.0:1734016147.943806:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335451392:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016147.943821:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.943830:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016147.943851:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016147.943852:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5556270 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 264 00000020:00010000:3.0:1734016147.943856:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016147.943857:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5556270 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 264 00000100:00100000:3.0:1734016147.943895:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dc850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335451392:12345-192.168.201.116@tcp:106: Request processed in 89us (308us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.943901:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12905 00010000:00010000:3.0:1734016147.954407:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.954429:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.954445:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.954451:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.954456:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016147.954795:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5556270, setting rss=5556270; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016147.954820:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.954827:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.954834:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.954841:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016147.954847:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.954857:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016147.954863:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.954869:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e77a5 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016147.957150:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334028928:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016147.963234:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335451520 00000100:00100000:3.0:1734016147.963248:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335451520 00000100:00100000:3.0:1734016147.963739:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12906 00000100:00100000:3.0:1734016147.963748:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335451520:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016147.963765:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.963775:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016147.963804:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016147.963806:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5556270 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 264 00000020:00010000:3.0:1734016147.963810:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016147.963811:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5556270 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 264 00000100:00100000:3.0:1734016147.964250:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335451520:12345-192.168.201.116@tcp:106: Request processed in 502us (1164us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.964259:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12906 00010000:00010000:3.0:1734016147.974272:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.974295:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.974310:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.974316:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.974321:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016147.974336:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5556270, setting rss=5556270; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016147.974349:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.974356:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.974362:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.974368:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016147.974375:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.974385:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016147.974390:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.974396:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e77ac lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016147.974422:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334029056:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016147.974814:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016147.974825:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334029184:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016147.991587:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335451648 00000100:00100000:3.0:1734016147.991597:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335451648 00000100:00100000:3.0:1734016147.991637:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12907 00000100:00100000:3.0:1734016147.991645:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335451648:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016147.991663:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.991673:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016147.991699:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016147.991700:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5556270 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 264 00000020:00010000:3.0:1734016147.991704:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016147.991705:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 5556270 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 264 00000100:00100000:3.0:1734016147.991749:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335451648:12345-192.168.201.116@tcp:106: Request processed in 105us (439us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016147.991754:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12907 00010000:00010000:3.0:1734016147.998427:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.998449:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.998461:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.998509:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.998516:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016147.998529:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5556270, setting rss=5556270; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016147.998540:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.998545:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.998552:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.998563:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367cc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016147.998569:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.998577:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016147.998581:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016147.998586:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e77b3 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016147.998628:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334029184:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.004127:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335451776 00000100:00100000:3.0:1734016148.004139:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335451776 00000100:00100000:3.0:1734016148.004200:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12908 00000100:00100000:3.0:1734016148.004209:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6479:x1818247335451776:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016148.004227:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016148.007665:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016148.007696:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:84392f61-a30b-4a4c-b6a5-f13f1ae7284e+5:6479:x1818247335451776:12345-192.168.201.116@tcp:104: Request processed in 3475us (3849us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.007711:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12908 00010000:00010000:0.0:1734016148.008193:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016148.008204:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.008216:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c18c0 already unused, calling callback (ffffffffa10571e0) 00010000:00010000:0.0:1734016148.008222:0:4750:0:(ldlm_request.c:1579:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.008230:0:4750:0:(ldlm_request.c:1589:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.008239:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.008269:0:4750:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781c900/0xa50fd972cb8e5f33 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115fb0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.008322:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 2/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.008327:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016148.008357:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.008364:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e7751 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117b78 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.008384:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016148.008386:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016148.008387:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016148.008396:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016148.008414:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334029568:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016148.012771:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3100 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334029440:192.168.201.116@tcp:36:fsx.0 00000080:00200000:3.0:1734016148.012827:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016148.012834:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016148.012856:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016148.012884:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1734016148.012900:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016148.012905:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4cf0, osc lock ffff8800a8a26be0, flags 0x0 00000008:00010000:3.0:1734016148.012908:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016148.012910:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4d30, osc lock ffff8800a8a26ab0, flags 0x0 00010000:00010000:3.0:1734016148.012946:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.012959:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.012998:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.013036:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334029696:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.013088:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4d80 going to sleep for 11 seconds 00000100:00100000:0.0:1734016148.013162:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334029568:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016148.022106:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335451904 00000100:00100000:3.0:1734016148.022123:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335451904 00000100:00100000:3.0:1734016148.022178:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12909 00000100:00100000:3.0:1734016148.022187:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335451904:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016148.022203:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366880/0xa50fd972cb8e7790 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5550080->5558271) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117bb7 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.022281:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016148.022288:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7684:x1818247335451904:12345-192.168.201.116@tcp:104: Request processed in 102us (1444us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.022295:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12909 00000100:00100000:3.0:1734016148.027704:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334029696:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016148.027747:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.027761:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.027770:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.028235:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335452032 00000100:00100000:3.0:1734016148.028244:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335452032 00000100:00100000:3.0:1734016148.028315:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12910 00000100:00100000:3.0:1734016148.028323:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6466:x1818247335452032:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016148.028334:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.028344:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.028354:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.028363:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.028371:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.028381:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.028416:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa34a400) 00000100:00100000:3.0:1734016148.028512:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6466:x1818247335452032:12345-192.168.201.116@tcp:105: Request processed in 188us (344us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.028520:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12910 00010000:00010000:0.0:1734016148.028752:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.028762:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.028786:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.028799:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 5556270, setting rss=5556270; kms=5556270, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.028812:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.028819:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.028847:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.028858:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.028866:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.028872:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.028879:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.028991:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334030080:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016148.029053:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016148.029059:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334029952:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016148.033107:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334029952:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016148.033133:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334030080:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016148.057615:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334030208:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016148.057644:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.057654:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.057667:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.057677:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.057687:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.057696:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.057700:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016148.057709:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78a2a00 x1818247334030208/t0(0) o101->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016164 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016148.057717:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016148.057730:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.057740:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016148.057808:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016148.057814:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e77c1 00000080:00010000:0.0:1734016148.057817:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016148.057820:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e77c1 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016148.057822:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.057827:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e77c1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117bfd expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016148.057853:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016148.057866:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016148.057887:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.057891:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55870, osc lock ffff8800a9f22e40, flags 0x400001002 00000008:00010000:0.0:1734016148.057899:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.057900:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a558b0, osc lock ffff8800a9f22130, flags 0x400001002 00010000:00010000:0.0:1734016148.057924:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.057930:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.057995:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e77cf lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.058001:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c33c0/0xa50fd972cb8e77cf lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.058043:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334030464:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016148.058117:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016148.058122:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334030336:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016148.081348:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.081368:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.081385:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.081394:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.081398:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.081412:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016148.081419:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.081423:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.081429:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.081435:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c18c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016148.081440:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.081450:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016148.081456:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.081461:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c18c0/0xa50fd972cb8e77c8 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.081819:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334030336:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.115346:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016148.115375:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334030848:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016148.120151:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334030848:192.168.201.116@tcp:22:fsx.0 00000100:00100000:0.0:1734016148.123015:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334030976:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016148.123054:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.123066:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.123080:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.123089:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.123099:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.123106:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016148.123112:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016148.123121:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78a0380 x1818247334030976/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016164 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016148.123130:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016148.123142:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.123152:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016148.123208:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016148.123214:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e77e4 00000080:00010000:0.0:1734016148.123218:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016148.123221:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e77e4 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016148.123224:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.123230:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016148.123266:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016148.123276:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016148.123295:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.123299:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a556f0, osc lock ffff8800a9f22980, flags 0x400001002 00000008:00010000:0.0:1734016148.123306:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.123307:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55730, osc lock ffff8800a9f22390, flags 0x400001002 00010000:00010000:0.0:1734016148.123321:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123332:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123342:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123349:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123364:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123371:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123379:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123384:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.123401:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7578183 i_blocks 264 00000020:00010000:0.0:1734016148.123405:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbec000 (ffff88012dc12248/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.123408:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbec000 (ffff88012dc12248/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.123410:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a556f0 (ffff88012dc12098/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.123413:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55730 (ffff88012dc12638/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.123415:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbec000 (ffff88012dc12248/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.123416:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a556f0 (ffff88012dc12248/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.123420:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55730 (ffff88012dc12248/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.123429:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016148.123446:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016148.123450:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016148.123453:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016148.123456:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.123458:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a556f0, osc lock ffff8800a9f22390, flags 0x400001002 00000008:00010000:0.0:1734016148.123461:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.123462:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55730, osc lock ffff8800a9f22980, flags 0x400001002 00010000:00010000:0.0:1734016148.123501:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123508:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123514:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123519:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123528:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123533:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123538:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.123542:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364900/0xa50fd972cb8e774a lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117b6a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.123550:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7578183 i_blocks 264 00000020:00010000:0.0:1734016148.123552:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbec000 (ffff88012dc12248/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.123554:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbec000 (ffff88012dc12248/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.123555:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a556f0 (ffff88012dc12638/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.123557:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55730 (ffff88012dc12098/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.123559:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbec000 (ffff88012dc12248/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.123561:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a556f0 (ffff88012dc12248/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.123563:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55730 (ffff88012dc12248/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.123567:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1734016148.130393:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335452800 00000100:00100000:3.0:1734016148.130406:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335452800 00000100:00100000:3.0:1734016148.130491:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12915 00000100:00100000:3.0:1734016148.130500:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:8c598caf-7141-4c19-8566-0b7731516db9+5:6479:x1818247335452800:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016148.130517:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e77e4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117c27 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016148.130605:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016148.130611:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:8c598caf-7141-4c19-8566-0b7731516db9+5:6479:x1818247335452800:12345-192.168.201.116@tcp:104: Request processed in 113us (335us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.130617:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12915 00000100:00100000:0.0:1734016148.140785:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fbb80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334031232:192.168.201.116@tcp:36:fsx.0 00000080:00200000:0.0:1734016148.140832:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016148.140839:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016148.140863:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016148.140886:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:0.0:1734016148.140901:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016148.140907:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a556f0, osc lock ffff8800a9f22980, flags 0x0 00000008:00010000:0.0:1734016148.140910:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016148.140911:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55730, osc lock ffff8800a9f22390, flags 0x0 00010000:00010000:0.0:1734016148.140946:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.140955:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.140990:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.141016:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0e00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334031488:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016148.142783:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55000 going to sleep for 11 seconds 00000100:00100000:3.0:1734016148.148617:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016148.154515:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016148.154536:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.154548:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa34a400 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016148.154555:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e77ba lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117bef expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.154583:0:4750:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016148.154606:0:4750:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000100:00100000:0.0:1734016148.154825:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0e00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334031488:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016148.154848:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.154862:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.154869:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.155214:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016148.155236:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334031616:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016148.167177:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335453056 00000100:00100000:3.0:1734016148.167192:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335453056 00000100:00100000:3.0:1734016148.167259:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12917 00000100:00100000:3.0:1734016148.167267:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:9876:x1818247335453056:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016148.167284:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.167296:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.167304:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.167315:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.167322:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.167333:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.167369:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c3180) 00000100:00100000:3.0:1734016148.167428:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:9876:x1818247335453056:12345-192.168.201.116@tcp:105: Request processed in 162us (2655us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.167435:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12917 00010000:00010000:0.0:1734016148.167672:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.167682:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.167709:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.167724:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.167736:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.167742:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.167773:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.167779:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.167807:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.167830:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0e00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334031744:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016148.167871:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55600 going to sleep for 11 seconds 00000100:00100000:3.0:1734016148.188422:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016148.188550:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334031616:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016148.200733:0:9558:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016148.200757:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335453312 00000100:00100000:3.0:1734016148.200765:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335453312 00000100:00100000:3.0:1734016148.200803:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12919 00000100:00100000:3.0:1734016148.200810:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f85b450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:9876:x1818247335453312:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016148.200824:0:9558:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.200833:0:9558:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.200840:0:9558:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.200850:0:9558:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.200859:0:9558:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.200868:0:9558:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.200883:0:9558:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c3a80) 00000100:00100000:3.0:1734016148.200931:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f85b450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:9876:x1818247335453312:12345-192.168.201.116@tcp:105: Request processed in 121us (348us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.200936:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12919 00000100:00100000:0.0:1734016148.201125:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0e00 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334031744:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016148.201142:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.201158:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.201168:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 3383879, setting rss=3383879; kms=3383879, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.201178:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.201184:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.201263:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016148.201275:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334032000:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016148.201328:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334032128:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016148.209048:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334032000:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016148.222885:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334032128:192.168.201.116@tcp:10:fsx.0 00000100:00100000:3.0:1734016148.249732:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334032512:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.256242:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334032640:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.266273:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335453568 00000100:00100000:3.0:1734016148.266280:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335453568 00000100:00100000:3.0:1734016148.266321:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12921 00000100:00100000:3.0:1734016148.266328:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335453568:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.266337:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.266345:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.266368:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.266370:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016147 ctime: 1734016148 blocks: 0 00000020:00010000:3.0:1734016148.266373:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016148.266374:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016147 ctime: 1734016148 blocks: 0 00000100:00100000:3.0:1734016148.266414:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7684:x1818247335453568:12345-192.168.201.116@tcp:106: Request processed in 86us (561us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.266420:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12921 00010000:00010000:3.0:1734016148.270030:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.270054:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.270068:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.270074:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.270082:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.270096:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.270108:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.270113:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.270119:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.270125:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367180 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.270129:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.270139:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.270143:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.270148:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa367180/0xa50fd972cb8e780e lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.270199:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334032640:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.278142:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335453824 00000100:00100000:3.0:1734016148.278158:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335453824 00000100:00100000:3.0:1734016148.278239:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12922 00000100:00100000:3.0:1734016148.278248:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335453824:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.278268:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.278279:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.278309:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.278312:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016147 ctime: 1734016148 blocks: 0 00000020:00010000:3.0:1734016148.278315:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016148.278316:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016147 ctime: 1734016148 blocks: 0 00000100:00100000:3.0:1734016148.278381:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335453824:12345-192.168.201.116@tcp:106: Request processed in 134us (2785us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.278389:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12922 00000100:00100000:3.0:1734016148.280455:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335453696 00000100:00100000:3.0:1734016148.280489:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335453696 00000100:00100000:3.0:1734016148.280534:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12923 00000100:00100000:3.0:1734016148.280540:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335453696:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.280552:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.280563:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.280595:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.280597:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2390667 mtime: 1734016148 atime: 1734016147 ctime: 1734016148 blocks: 216 00000020:00010000:3.0:1734016148.280600:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016148.280600:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 2390667 mtime: 1734016148 atime: 1734016147 ctime: 1734016148 blocks: 216 00000100:00100000:3.0:1734016148.280643:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335453696:12345-192.168.201.116@tcp:106: Request processed in 103us (861us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.280649:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12923 00010000:00010000:3.0:1734016148.290910:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.290932:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.290946:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.290953:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.290959:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.290976:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.290984:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.290995:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291002:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291009:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.291027:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291042:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.291048:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291054:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa364b40/0xa50fd972cb8e781c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.291110:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334032896:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016148.291131:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291137:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291143:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291149:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291154:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.291160:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2390667, setting rss=2390667; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.291186:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291192:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291197:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291201:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.291204:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291209:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.291212:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.291218:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367840/0xa50fd972cb8e7815 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.291234:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334032768:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.300815:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335453952 00000100:00100000:3.0:1734016148.300826:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335453952 00000100:00100000:3.0:1734016148.300876:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12924 00000100:00100000:3.0:1734016148.300884:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335453952:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016148.300901:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c3180/0xa50fd972cb8e77eb lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117c35 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.300983:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016148.300990:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335453952:12345-192.168.201.116@tcp:104: Request processed in 106us (661us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.300996:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12924 00000100:00100000:3.0:1734016148.310459:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000008:00010000:3.0:1734016148.325636:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7823 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1929216->1937407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c7b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.325756:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1929216 1933311) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7823 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1929216->1937407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c7b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016148.325944:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982d80 - object 0x240000400:6 offset 0<>2392064 00000008:00010000:3.0:1734016148.384197:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7823 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1929216->1937407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c7b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.386947:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7823 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1929216->1937407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c7b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.387010:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334033408:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016148.390569:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334033536:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016148.396141:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e782a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.396167:0:1984:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e782a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117c82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.396184:0:1984:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e782a lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117c82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.396195:0:1984:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e782a lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117c82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.396211:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 2390667, setting rss=2390667; kms=2390667, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e782a lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117c82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.396238:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e782a lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117c82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.396244:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e782a lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117c82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.396250:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e782a lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117c82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.396256:0:1984:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e782a lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117c82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.396276:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334033536:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.396935:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334033664:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.401297:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016148.401318:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335454208 00000100:00100000:3.0:1734016148.401326:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335454208 00000100:00100000:3.0:1734016148.401370:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12926 00000100:00100000:3.0:1734016148.401378:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dd050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335454208:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.401394:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.401405:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.401432:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.401434:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016147 ctime: 1734016148 blocks: 0 00000020:00010000:3.0:1734016148.401437:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016148.401438:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016147 ctime: 1734016148 blocks: 0 00000100:00100000:3.0:1734016148.401520:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dd050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335454208:12345-192.168.201.116@tcp:106: Request processed in 142us (357us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.401527:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12926 00010000:00010000:3.0:1734016148.405154:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.405178:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.405818:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.405826:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.405832:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.405847:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.405867:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.409525:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.409549:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.409555:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.409563:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.409582:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.409588:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.409594:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e7831 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.409647:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334033664:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.425592:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335454336 00000100:00100000:3.0:1734016148.425603:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335454336 00000100:00100000:3.0:1734016148.425642:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335454464 00000100:00100000:3.0:1734016148.425644:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335454464 00000100:00100000:3.0:1734016148.425658:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12927 00000100:00100000:3.0:1734016148.425665:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335454336:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016148.425685:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e782a lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117c82 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.425767:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016148.425773:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335454336:12345-192.168.201.116@tcp:104: Request processed in 108us (344us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.425779:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12927 00000100:00100000:3.0:1734016148.425794:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12928 00000100:00100000:3.0:1734016148.425798:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335454464:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016148.425806:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e7823 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1929216->1937407) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117c7b expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.425838:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016148.425842:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335454464:12345-192.168.201.116@tcp:104: Request processed in 43us (330us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.425847:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12928 00010000:00010000:0.0:1734016148.431122:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016148.431147:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016148.431182:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334033920:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016148.434235:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016148.434257:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334034048:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016148.439271:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334033792:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016148.439309:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3055616->3063807] (req 3055616->3063807) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.439321:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3055616->3063807] (req 3055616->3063807) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.439329:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [3055616->3063807] (req 3055616->3063807) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.440506:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.440514:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.440533:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.440549:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 2390667, setting rss=2390667; kms=2390667, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.440561:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.440566:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.440594:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 2390667 i_blocks 216 00000020:00010000:0.0:1734016148.445954:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.445958:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.445964:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811b30 (ffff88012dc123f8/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016148.445974:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.445988:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016148.446006:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.446009:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811b30 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.446036:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016148.446108:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016148.446116:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016148.446137:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016148.446153:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.446163:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016148.446174:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016148.446176:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e77f9 00000080:00010000:0.0:1734016148.446183:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e77f9 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016148.446185:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.446190:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016148.446208:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016148.446223:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016148.446249:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.446252:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55630, osc lock ffff8800a9f22720, flags 0x400001002 00000008:00010000:0.0:1734016148.446260:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.446262:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55670, osc lock ffff8800a9f225f0, flags 0x400001002 00010000:00010000:0.0:1734016148.446286:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e783f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.446294:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2400/0xa50fd972cb8e783f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.446451:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.446528:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.446542:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.446553:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.449511:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335454720 00000100:00100000:3.0:1734016148.449521:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335454720 00000100:00100000:3.0:1734016148.449568:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12930 00000100:00100000:3.0:1734016148.449575:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335454720:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.449590:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.449600:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.449627:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.449629:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2390667 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 232 00000020:00010000:3.0:1734016148.449632:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016148.449633:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 2390667 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 232 00000100:00100000:3.0:1734016148.449681:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335454720:12345-192.168.201.116@tcp:106: Request processed in 107us (321us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.449687:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12930 00000080:00200000:0.0:1734016148.452521:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 2390667 i_blocks 232 00000020:00010000:0.0:1734016148.452529:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.452532:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.452538:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55630 (ffff88012dc123f8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.452542:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55670 (ffff8800a8a54bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.452546:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.452548:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55630 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.452551:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55670 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.452564:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016148.452588:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016148.452594:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016148.452605:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016148.452625:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.452628:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55630, osc lock ffff8800a9f225f0, flags 0x400001002 00000008:00010000:0.0:1734016148.452634:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.452636:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55670, osc lock ffff8800a9f22720, flags 0x400001002 00010000:00010000:0.0:1734016148.452662:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e7846 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.452670:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c2d00/0xa50fd972cb8e7846 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.452732:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.452741:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.452750:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.452757:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.455693:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335454848 00000100:00100000:3.0:1734016148.455704:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335454848 00000100:00100000:3.0:1734016148.455746:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12931 00000100:00100000:3.0:1734016148.455754:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335454848:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.455769:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.455779:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.455803:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.455805:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2390667 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 232 00000020:00010000:3.0:1734016148.455808:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016148.455809:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 2390667 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 232 00000100:00100000:3.0:1734016148.455855:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335454848:12345-192.168.201.116@tcp:106: Request processed in 102us (343us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.455862:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12931 00000080:00200000:0.0:1734016148.461113:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 2390667 i_blocks 232 00000020:00010000:0.0:1734016148.461122:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.461126:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.461132:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55630 (ffff8800a8a54bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.461137:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55670 (ffff88012dc123f8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.461139:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.461141:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55630 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.461145:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55670 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.461161:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016148.461176:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 2390667 00000080:00200008:0.0:1734016148.461268:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 25182, type: SEEK_SET 00000080:00200008:0.0:1734016148.461275:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 25182, type: SEEK_SET, rc = 25182 00000080:00200008:0.0:1734016148.461286:0:10406:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 25182, count: 215 00000080:00200000:0.0:1734016148.461293:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016148.461307:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016148.461347:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 6 00000080:00200000:0.0:1734016148.461356:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 25182, bytes: 215 00000080:00200000:0.0:1734016148.461361:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016148.461368:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016148.461370:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [25182, 25397) 0 0 00000080:00200000:0.0:1734016148.461372:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016148.461393:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [25182, 25397) 25397, 215 00020000:00200000:0.0:1734016148.461399:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016148.461421:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [6, 6] 00000008:00010000:0.0:1734016148.461439:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016148.461443:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369990b0, osc lock ffff8800a9f22720, flags 0x0 00010000:00010000:0.0:1734016148.461696:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.508206:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334034816:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.512074:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335454976 00000100:00100000:3.0:1734016148.512086:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335454976 00000100:00100000:3.0:1734016148.512228:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12932 00000100:00100000:3.0:1734016148.512235:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335454976:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.512251:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.512261:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.512288:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.512289:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000020:00010000:3.0:1734016148.512293:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016148.512294:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000100:00100000:3.0:1734016148.512548:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335454976:12345-192.168.201.116@tcp:106: Request processed in 313us (582us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.512556:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12932 00010000:00010000:3.0:1734016148.515392:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.515413:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.515433:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.515441:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.515447:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.515487:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.515509:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.515516:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.515522:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.515528:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366d00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.515535:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.515545:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.515549:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.515556:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366d00/0xa50fd972cb8e784d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.515901:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334034816:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.515922:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016148.515936:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334034944:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.519901:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016148.519923:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335455104 00000100:00100000:3.0:1734016148.519932:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335455104 00000100:00100000:3.0:1734016148.519972:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12933 00000100:00100000:3.0:1734016148.519978:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2de050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335455104:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.519994:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.520006:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.520050:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.520052:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000020:00010000:3.0:1734016148.520054:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016148.520055:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000100:00100000:3.0:1734016148.520101:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2de050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335455104:12345-192.168.201.116@tcp:106: Request processed in 123us (321us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.520107:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12933 00010000:00010000:3.0:1734016148.523789:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.523811:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.523825:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.523831:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.523836:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.523850:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.523877:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.523888:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.523895:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.523901:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.523905:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.523916:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.523920:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.523925:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e7854 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.523967:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334034944:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.524421:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334035072:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.534803:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335455232 00000100:00100000:3.0:1734016148.534817:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335455232 00000100:00100000:3.0:1734016148.534878:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12934 00000100:00100000:3.0:1734016148.534886:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335455232:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.534902:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.534913:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.534943:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.534945:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000020:00010000:3.0:1734016148.534948:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016148.534949:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000100:00100000:3.0:1734016148.535001:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335455232:12345-192.168.201.116@tcp:106: Request processed in 116us (298us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.535008:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12934 00010000:00010000:3.0:1734016148.549190:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.549211:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.549228:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.549234:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.549240:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.549257:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.549283:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.549289:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.549295:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.549301:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.549307:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.549319:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.549325:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.549330:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e785b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.549378:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334035072:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.549922:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016148.549932:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334035200:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.553121:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335455360 00000100:00100000:3.0:1734016148.553131:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335455360 00000100:00100000:3.0:1734016148.553172:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12935 00000100:00100000:3.0:1734016148.553179:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335455360:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.553195:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.553206:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.553234:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.553236:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000020:00010000:3.0:1734016148.553239:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016148.553240:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000100:00100000:3.0:1734016148.553288:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335455360:12345-192.168.201.116@tcp:106: Request processed in 108us (252us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.553295:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12935 00010000:00010000:3.0:1734016148.556372:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.556394:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.556407:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.556413:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.556419:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.556439:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.556453:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.556458:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.556501:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.556507:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.556512:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.556522:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.556527:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.556533:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e7862 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.556575:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334035200:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.563880:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335455488 00000100:00100000:3.0:1734016148.563892:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335455488 00000100:00100000:3.0:1734016148.563931:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12936 00000100:00100000:3.0:1734016148.563938:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335455488:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016148.563957:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa367cc0/0xa50fd972cb8e7838 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3055616->3063807) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117c90 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.564049:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016148.564055:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335455488:12345-192.168.201.116@tcp:104: Request processed in 117us (284us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.564061:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12936 00000100:00100000:3.0:1734016148.569956:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334035328:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016148.569988:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1589248->1597439] (req 1589248->1597439) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.570001:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1589248->1597439] (req 1589248->1597439) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.570008:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1589248->1597439] (req 1589248->1597439) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.581535:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335455616 00000100:00100000:3.0:1734016148.581547:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335455616 00000100:00100000:3.0:1734016148.581647:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12937 00000100:00100000:3.0:1734016148.581655:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:9876:x1818247335455616:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016148.581673:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1589248->1597439] (req 1589248->1597439) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.581684:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [1589248->1597439] (req 1589248->1597439) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.581699:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.581714:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.581723:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.581733:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.581917:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366f40) 00010000:00010000:0.0:1734016148.582051:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.582062:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.582090:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.582103:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 2390667, setting rss=2390667; kms=2390667, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.582120:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.582127:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.582155:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 2390667 i_blocks 232 00000100:00100000:0.0:1734016148.582233:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016148.582249:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334035584:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016148.582312:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:9876:x1818247335455616:12345-192.168.201.116@tcp:105: Request processed in 656us (963us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.582319:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12937 00000100:00100000:0.0:1734016148.590059:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334035584:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016148.595731:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334035712:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.598612:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016148.598632:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334035840:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.600105:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335455744 00000100:00100000:3.0:1734016148.600113:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335455744 00000100:00100000:3.0:1734016148.600157:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12938 00000100:00100000:3.0:1734016148.600173:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335455744:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.600186:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.600197:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.600225:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.600227:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 6088193 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 232 00000020:00010000:3.0:1734016148.600230:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016148.600232:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 6088193 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 232 00000100:00100000:3.0:1734016148.600288:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335455744:12345-192.168.201.116@tcp:106: Request processed in 123us (250us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.600295:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12938 00000100:00100000:3.0:1734016148.607205:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335455872 00000100:00100000:3.0:1734016148.607218:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335455872 00000100:00100000:3.0:1734016148.607276:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12939 00000100:00100000:3.0:1734016148.607287:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335455872:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.607306:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.607317:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.607350:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.607353:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000020:00010000:3.0:1734016148.607356:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016148.607357:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000100:00100000:3.0:1734016148.607416:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335455872:12345-192.168.201.116@tcp:106: Request processed in 130us (965us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.607424:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12939 00010000:00010000:3.0:1734016148.609358:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.609375:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.609388:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.609395:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.609402:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.609416:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6088193, setting rss=6088193; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.609424:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.609430:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.609437:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.609445:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.609451:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.609462:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.609489:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.609496:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa366640/0xa50fd972cb8e7870 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.610090:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334035712:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016148.611668:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 232 00000020:00010000:0.0:1734016148.611674:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.611677:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.611683:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee77b0 (ffff88012dc123f8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.611687:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee77f0 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.611690:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.611692:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee77b0 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.611696:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee77f0 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.611709:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016148.611732:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016148.611739:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016148.611748:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016148.611768:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.611773:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55630, osc lock ffff8800a9f225f0, flags 0x400001002 00000008:00010000:0.0:1734016148.611781:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.611783:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55670, osc lock ffff8800a9f22260, flags 0x400001002 00010000:00010000:0.0:1734016148.611807:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.611816:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.611878:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.611885:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.611916:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334035968:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016148.611981:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016148.611987:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334036096:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.615144:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016148.616484:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016148.616548:0:9611:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335456128 00000100:00100000:0.0:1734016148.616553:0:9611:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335456128 00000100:00100000:0.0:1734016148.616591:0:9611:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 232 00000100:00100000:0.0:1734016148.616597:0:9611:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8216:x1818247335456128:12345-192.168.201.116@tcp:106: 00010000:00010000:0.0:1734016148.616605:0:9611:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.616613:0:9611:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016148.616628:0:9611:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 2, [0x200000401:0xc:0x0] hash] 00000020:00010000:0.0:1734016148.616630:0:9611:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000020:00010000:0.0:1734016148.616633:0:9611:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:0.0:1734016148.616634:0:9611:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 0 00000100:00100000:0.0:1734016148.616656:0:9611:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7983100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:8216:x1818247335456128:12345-192.168.201.116@tcp:106: Request processed in 59us (173us total) trans 0 rc 0/0 00000100:00100000:0.0:1734016148.616663:0:9611:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 232 00010000:00010000:0.0:1734016148.621141:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621168:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621185:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621192:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621199:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.621220:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016148.621230:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621236:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621243:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621251:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c2ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016148.621259:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621272:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016148.621279:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621286:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c2ac0/0xa50fd972cb8e7885 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.621383:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334036096:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016148.621403:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621410:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621417:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621422:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621428:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.621436:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6088193, setting rss=6088193; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016148.621457:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621480:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621487:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621492:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0fc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016148.621495:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621502:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016148.621505:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.621511:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0fc0/0xa50fd972cb8e787e lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.621530:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334035968:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.626450:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016148.626935:0:4749:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016148.626943:0:4749:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.626956:0:4749:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c3a80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016148.626961:0:4749:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.626983:0:4749:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016148.626998:0:4749:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016148.627053:0:4749:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.627060:0:4749:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016148.627093:0:4749:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.627100:0:4749:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e77f2 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117c3c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.627130:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016148.627131:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016148.627133:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1734016148.634582:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016148.642852:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016148.643535:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334036352:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016148.649102:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016148.649117:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334036608:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.652733:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016148.664929:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 233 00000020:00010000:0.0:1734016148.664937:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.664940:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.664946:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7f30 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.664950:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7f70 (ffff88012dc123f8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.664953:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.664955:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7f30 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.664959:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7f70 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.664971:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016148.664983:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10282497 00000080:00200008:0.0:1734016148.665043:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 5252207, type: SEEK_SET 00000080:00200008:0.0:1734016148.665047:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 5252207, type: SEEK_SET, rc = 5252207 00000080:00200008:0.0:1734016148.665056:0:10406:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 5252207, count: 1832 00000080:00200000:0.0:1734016148.665062:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016148.665071:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016148.665108:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 1282 00000080:00200000:0.0:1734016148.665116:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 5252207, bytes: 1832 00000080:00200000:0.0:1734016148.669119:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016148.669139:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5ce10[0x4, 2, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016148.669142:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [5252207, 5254039) 0 0 00000080:00200000:0.0:1734016148.669149:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016148.669173:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [5252207, 5254039) 5254039, 1832 00020000:00200000:0.0:1734016148.669177:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016148.669199:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [1282, 1282] 00000008:00010000:0.0:1734016148.669222:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016148.669227:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999a30, osc lock ffff8800a9f22260, flags 0x0 00010000:00010000:0.0:1734016148.669261:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.669271:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1056768->1060863] (req 1056768->1060863) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.669316:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1056768->1060863] (req 1056768->1060863) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.669346:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334036736:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016148.669430:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55c00 going to sleep for 11 seconds 00000100:00100000:3.0:1734016148.677225:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016148.682432:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016148.682453:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334036864:192.168.201.116@tcp:4:fsx.0 00000100:00100000:0.0:1734016148.691431:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334036736:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016148.691508:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1056768->1060863] (req 1056768->1060863) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.691520:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1056768->1060863] (req 1056768->1060863) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.691529:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [1056768->1060863] (req 1056768->1060863) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.695235:0:1981:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff88012d4859c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e788c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3350528->3362815) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117ce4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.695290:0:1981:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff88012d4859c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a781c240/0xa50fd972cb8e788c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3350528->3362815) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117ce4 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.695309:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334036864:192.168.201.116@tcp:4:fsx.0 00000100:00100000:3.0:1734016148.706653:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016148.710003:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.710025:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.710051:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.710065:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 3359509, setting rss=3359509; kms=3359509, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.710078:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.710083:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.710098:0:10406:0:(vvp_io.c:840:vvp_io_read_start()) f16b.sanityn: read [5252207, 5254039) 00000080:00200000:0.0:1734016148.710130:0:10406:0:(vvp_io.c:899:vvp_io_read_start()) count:1832 iocb pos:5252207 00010000:00010000:0.0:1734016148.710188:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.710199:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.710207:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 6/1,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.710218:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 6/1,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.710222:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.748051:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.748070:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.764650:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334037504:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016148.820821:0:1981:0:(osc_cache.c:824:osc_extent_finish()) ### no dlm lock: extent: ffff8800a8a55d80 00000008:00010000:0.0:1734016148.821156:0:1981:0:(osc_cache.c:484:__osc_extent_remove()) ### no dlm lock: extent: ffff8800a8a55d80 00000100:00100000:0.0:1734016148.821182:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334037504:192.168.201.116@tcp:3:fsx.0 00000020:00010000:0.0:1734016148.824276:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.824280:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.824285:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff880136999a30 (ffff88012dc123f8/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016148.824294:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824305:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016148.824326:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.824328:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff880136999a30 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.824336:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 1832 bytes 00000080:00200000:0.0:1734016148.824352:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016148.824357:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 1832, restart: 0 00000080:00200000:0.0:1734016148.824362:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 1832 00000080:00200008:0.0:1734016148.824388:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016148.824395:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016148.824414:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016148.824432:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.824443:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016148.824454:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016148.824457:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e77f9 00000080:00010000:0.0:1734016148.824494:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e77f9 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016148.824498:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016148.824504:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016148.824525:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016148.824537:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016148.824554:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.824557:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55db0, osc lock ffff8800a9f22260, flags 0x400001002 00000008:00010000:0.0:1734016148.824561:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.824562:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55df0, osc lock ffff8800a9f225f0, flags 0x400001002 00010000:00010000:0.0:1734016148.824575:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824584:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824593:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824599:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824611:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824618:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824625:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824631:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.824649:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 256 00000020:00010000:0.0:1734016148.824653:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.824655:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.824657:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55db0 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.824660:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55df0 (ffff88012dc123f8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.824662:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.824664:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55db0 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.824667:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55df0 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.824672:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016148.824682:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016148.824686:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016148.824688:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016148.824693:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.824694:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55db0, osc lock ffff8800a9f225f0, flags 0x400001002 00000008:00010000:0.0:1734016148.824697:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.824698:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55df0, osc lock ffff8800a9f22260, flags 0x400001002 00010000:00010000:0.0:1734016148.824703:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824710:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824718:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824724:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824734:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824741:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824748:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824754:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.824762:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 256 00000020:00010000:0.0:1734016148.824765:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.824767:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.824769:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55db0 (ffff88012dc123f8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.824771:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55df0 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.824773:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.824775:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55db0 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.824777:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55df0 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.824782:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016148.824789:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10282497 00000080:00200000:0.0:1734016148.824839:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:0.0:1734016148.824843:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:0.0:1734016148.824850:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016148.824858:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2016, 2016] 00000008:00010000:0.0:1734016148.824865:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016148.824867:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff880136999a30, osc lock ffff8800a9f22260, flags 0x0 00010000:00010000:0.0:1734016148.824873:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800a9dd5400 type 11 mode 2 res 10737419264/8 (4063232 4067327) 00010000:00010000:0.0:1734016148.824884:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824892:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [4063232->4067327] (req 4063232->4067327) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.824926:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [4063232->4067327] (req 4063232->4067327) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.824951:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334037632:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016148.825459:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55d80 going to sleep for 11 seconds 00000100:00100000:3.0:1734016148.831677:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335457024 00000100:00100000:3.0:1734016148.831687:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335457024 00000100:00100000:3.0:1734016148.832220:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12947 00000100:00100000:3.0:1734016148.832228:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335457024:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016148.832245:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.832331:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016148.832338:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335457024:12345-192.168.201.116@tcp:104: Request processed in 111us (1210us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.832345:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12947 00010000:00010000:0.0:1734016148.832600:0:4749:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016148.832608:0:4749:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.832616:0:4749:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c2f40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016148.832621:0:4749:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.832640:0:4749:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016148.836059:0:4749:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016148.836139:0:4749:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.836149:0:4749:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016148.836190:0:4749:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.836202:0:4749:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c2f40/0xa50fd972cb8e78a1 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1056768->1060863) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117cf9 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.836217:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016148.836219:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016148.836222:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016148.836238:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016148.836260:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334037760:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016148.843112:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016148.850444:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334037760:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016148.855069:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334038016:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.855362:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334038144:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.862310:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12949 00000100:00100000:3.0:1734016148.862320:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335457280:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.862336:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.862347:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.862380:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1026, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.862382:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 6088193 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 232 00000020:00010000:3.0:1734016148.862385:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016148.862386:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 6088193 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 232 00000100:00100000:3.0:1734016148.862445:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335457280:12345-192.168.201.116@tcp:106: Request processed in 126us (1349us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.862452:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12949 00000100:00100000:3.0:1734016148.862482:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12950 00000100:00100000:3.0:1734016148.862486:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335457408:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016148.862494:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.862501:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016148.862512:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1026, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016148.862514:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3359509 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 24 00000020:00010000:3.0:1734016148.862516:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016148.862517:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3359509 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 24 00000100:00100000:3.0:1734016148.862534:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335457408:12345-192.168.201.116@tcp:106: Request processed in 47us (1245us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016148.862539:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12950 00010000:00010000:3.0:1734016148.866829:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.866853:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.866868:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.866875:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.866882:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.866902:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6088193, setting rss=6088193; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.866910:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.866916:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.866924:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.866930:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.866936:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.866950:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.866956:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.866963:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e78af lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.867004:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334038016:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016148.870644:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.870668:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.870679:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.870685:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.870691:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016148.870704:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3359509, setting rss=3359509; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016148.870716:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.870722:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.870728:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.870734:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016148.870739:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.870754:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016148.870760:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.870766:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e78b6 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.870796:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334038144:192.168.201.116@tcp:101:fsx.0 00000080:00200000:3.0:1734016148.884158:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 256 00000020:00010000:3.0:1734016148.884166:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:3.0:1734016148.884169:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:3.0:1734016148.884176:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7ab0 (ffff88012dc12908/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016148.884179:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7af0 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:3.0:1734016148.884181:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:3.0:1734016148.884183:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7ab0 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:3.0:1734016148.884187:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7af0 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:3.0:1734016148.884201:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1734016148.884213:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10282497 00000080:00200000:3.0:1734016148.884284:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016148.884288:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016148.884304:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016148.884317:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [131, 133] 00000008:00010000:3.0:1734016148.884333:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016148.884337:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b58ba4b0, osc lock ffff8800a8a26720, flags 0x0 00010000:00010000:3.0:1734016148.884363:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.884375:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [536576->548863] (req 536576->548863) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.884429:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [536576->548863] (req 536576->548863) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.884456:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334038528:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.884919:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1734016148.888371:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016148.889434:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334038656:192.168.201.116@tcp:4:fsx.0 00000100:00100000:3.0:1734016148.894522:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334038528:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016148.894555:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [536576->548863] (req 536576->548863) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d23 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.894566:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [536576->548863] (req 536576->548863) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d23 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016148.894573:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [536576->548863] (req 536576->548863) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d23 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.895745:0:1982:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.896178:0:1982:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7600 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366f40/0xa50fd972cb8e7869 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1589248->1597439) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117cc1 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.896200:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334038656:192.168.201.116@tcp:4:fsx.0 00000100:00100000:0.0:1734016148.902900:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334038784:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016148.913819:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016148.914351:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 536576->548863) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d23 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.914360:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 536576->548863) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d23 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.914390:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 536576->548863) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d23 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.914403:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 6088193, setting rss=6088193; kms=6088193, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 536576->548863) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117d23 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.914414:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 536576->548863) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d23 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.914420:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa34a400/0xa50fd972cb8e78cb lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 536576->548863) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d23 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.914446:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 264 00000100:00100000:0.0:1734016148.914560:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016148.914572:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334038912:192.168.201.116@tcp:22:fsx.0 00000100:00100000:0.0:1734016148.917131:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334038784:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016148.926306:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334038912:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016148.926729:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016148.926757:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334039040:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.929903:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016148.941445:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 264 00000020:00010000:0.0:1734016148.941451:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.941453:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.941459:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee79f0 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.941462:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7a30 (ffff88012dc122d8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.941492:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.941494:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee79f0 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.941498:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7a30 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.941508:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016148.941529:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016148.941534:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016148.941547:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016148.941564:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.941567:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55870, osc lock ffff8800a9f225f0, flags 0x400001002 00000008:00010000:0.0:1734016148.941572:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016148.941573:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a558b0, osc lock ffff8800a9f22720, flags 0x400001002 00010000:00010000:0.0:1734016148.941592:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.941600:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.941649:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.941667:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334039168:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016148.941735:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.941744:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.941750:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.952066:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016148.962435:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.962450:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.962462:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.962492:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.962498:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.962510:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6088193, setting rss=6088193; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016148.962527:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.962532:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.962538:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.962542:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1b00 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016148.962548:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.962556:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016148.962561:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.962566:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e78d9 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.962596:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334039168:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016148.966391:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016148.976778:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 264 00000020:00010000:0.0:1734016148.976788:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_release():233 00000020:00010000:0.0:1734016148.976791:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.976797:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7e70 (ffff88012dc122d8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.976802:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7eb0 (ffff8800b3af1c68/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016148.976806:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedda8 (ffff88012dc12128/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.976808:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7e70 (ffff88012dc12128/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016148.976812:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7eb0 (ffff88012dc12128/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016148.976824:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016148.976858:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 2097 00000080:00200000:0.0:1734016148.976867:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 8590130, bytes: 4433 00000080:00200000:0.0:1734016148.976871:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016148.976878:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016148.976880:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [8590130, 8594563) 0 0 00000080:00200000:0.0:1734016148.976883:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016148.976902:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 2 chunk: [8590130, 8594563) 8594563, 4433 00020000:00200000:0.0:1734016148.976906:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016148.976921:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [2097, 2098] 00000008:00010000:0.0:1734016148.976935:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016148.976938:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369991b0, osc lock ffff8800a9f22390, flags 0x0 00010000:00010000:0.0:1734016148.976962:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.976971:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4395008->4403199] (req 4395008->4403199) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.977015:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4395008->4403199] (req 4395008->4403199) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016148.977038:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2300 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334039424:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016148.977099:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55480 going to sleep for 11 seconds 00000100:00100000:3.0:1734016148.984286:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:0.0:1734016148.988822:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334039552:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016148.996214:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2300 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334039424:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016148.996241:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4395008->4403199] (req 4395008->4403199) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.996252:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4395008->4403199] (req 4395008->4403199) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.996260:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [4395008->4403199] (req 4395008->4403199) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016148.997150:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016148.997562:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.997574:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.997591:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016148.997603:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 6088193, setting rss=6088193; kms=6088193, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.997615:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.997621:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016148.997634:0:10406:0:(vvp_io.c:840:vvp_io_read_start()) f16b.sanityn: read [8590130, 8594563) 00000080:00200000:0.0:1734016148.997655:0:10406:0:(vvp_io.c:899:vvp_io_read_start()) count:4433 iocb pos:8590130 00010000:00010000:0.0:1734016148.997692:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016148.997702:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.004096:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334039552:192.168.201.116@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1734016149.013016:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.013030:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.013060:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.013072:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.013079:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.013084:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.013089:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.028971:0:1981:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff88012d485900 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.028987:0:1981:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff88012d4859c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.029037:0:1981:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4395008 4399103) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016149.029192:0:1981:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e2300 - object 0x240000400:6 offset 0<>4399104 00000008:00010000:0.0:1734016149.029204:0:1981:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff88012d4859c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.029219:0:1981:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4190208 4194303) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016149.029231:0:1981:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e0700 - object 0x240000400:6 offset 4190208<>4194304 00000100:00100000:0.0:1734016149.029252:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334039936:192.168.201.116@tcp:3:fsx.0 00000100:00100000:0.0:1734016149.029395:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016149.029401:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334040064:192.168.201.116@tcp:3:fsx.0 00010000:00010000:0.0:1734016149.035570:0:1982:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016149.035745:0:1982:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e3b80 - object 0x280000400:8 offset 0<>3362816 00000100:00100000:0.0:1734016149.035792:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334040320:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016149.049267:0:1981:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff88012d4859c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.049292:0:1981:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff88012d4859c0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.049313:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334040064:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016149.229427:0:1981:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff88012d485900 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.242086:0:1981:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff88012d485900 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.242123:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334039936:192.168.201.116@tcp:3:fsx.0 00000020:00010000:0.0:1734016149.243046:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016149.243049:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.243053:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801369991b0 (ffff88012dc12128/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016149.243060:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.243067:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016149.243080:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.243081:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801369991b0 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016149.243087:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 4433 bytes 00000080:00200000:0.0:1734016149.243098:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016149.243101:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 4433, restart: 0 00000080:00200000:0.0:1734016149.243105:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 4433 00000080:00200008:0.0:1734016149.243127:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016149.243132:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016149.243153:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016149.243163:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.243172:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016149.243180:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016149.243182:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e7800 00000080:00010000:0.0:1734016149.243186:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7800 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016149.243189:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.243193:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016149.243209:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016149.243219:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016149.243235:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.243238:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a554b0, osc lock ffff8800a9f22390, flags 0x400001002 00000008:00010000:0.0:1734016149.243242:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.243243:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a554f0, osc lock ffff8800a9f22980, flags 0x400001002 00010000:00010000:0.0:1734016149.243259:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.243265:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.243846:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.243852:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.243885:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334040704:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016149.243937:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016149.243943:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334040832:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016149.249960:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.249976:0:1981:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.249988:0:1981:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.250006:0:1981:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.250019:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 6088193, setting rss=6088193; kms=6088193, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.250029:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.250035:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.250044:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.250049:0:1981:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.250061:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334040704:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.250124:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335458688 00000100:00100000:3.0:1734016149.250130:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335458688 00000100:00100000:3.0:1734016149.250171:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12960 00000100:00100000:3.0:1734016149.250177:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335458688:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016149.250185:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.250193:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016149.250211:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1848, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016149.250213:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3359509 mtime: 1734016148 atime: 1734016149 ctime: 1734016148 blocks: 24 00000020:00010000:3.0:1734016149.250216:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016149.250217:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3359509 mtime: 1734016148 atime: 1734016149 ctime: 1734016148 blocks: 24 00000100:00100000:3.0:1734016149.250254:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7683:x1818247335458688:12345-192.168.201.116@tcp:106: Request processed in 77us (413us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.250260:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12960 00010000:00010000:0.0:1734016149.251861:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.251871:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.251881:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.251886:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.251892:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.251902:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3359509, setting rss=3359509; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016149.251911:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.251917:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.251923:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.251927:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c21c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.251932:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.251939:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016149.251942:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.251947:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c21c0/0xa50fd972cb8e78f5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.251974:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334040832:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.267764:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335458816 00000100:00100000:3.0:1734016149.267773:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335458816 00000100:00100000:3.0:1734016149.267812:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12961 00000100:00100000:3.0:1734016149.267819:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335458816:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016149.267833:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.267842:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016149.267865:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1848, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016149.267867:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3359509 mtime: 1734016148 atime: 1734016149 ctime: 1734016148 blocks: 24 00000020:00010000:3.0:1734016149.267870:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016149.267871:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3359509 mtime: 1734016148 atime: 1734016149 ctime: 1734016148 blocks: 24 00000100:00100000:3.0:1734016149.269049:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335458816:12345-192.168.201.116@tcp:106: Request processed in 1218us (1486us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.269059:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12961 00010000:00010000:3.0:1734016149.280200:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.280218:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.280230:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.280236:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.280242:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016149.280254:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3359509, setting rss=3359509; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016149.280275:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.280286:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.280292:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.280298:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365f80 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016149.280303:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.280314:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016149.280319:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.280325:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365f80/0xa50fd972cb8e78fc lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.280365:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334040960:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.286630:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335458944 00000100:00100000:3.0:1734016149.286639:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335458944 00000100:00100000:3.0:1734016149.286684:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12962 00000100:00100000:3.0:1734016149.286689:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335458944:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016149.286702:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.286716:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016149.286738:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1848, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016149.286740:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3359509 mtime: 1734016148 atime: 1734016149 ctime: 1734016148 blocks: 24 00000020:00010000:3.0:1734016149.286743:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016149.286744:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3359509 mtime: 1734016148 atime: 1734016149 ctime: 1734016148 blocks: 24 00000100:00100000:3.0:1734016149.286780:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335458944:12345-192.168.201.116@tcp:106: Request processed in 90us (220us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.286787:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12962 00010000:00010000:3.0:1734016149.295847:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.295865:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.295878:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.296598:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.296609:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016149.296624:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3359509, setting rss=3359509; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016149.296646:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.296653:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.296659:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.296665:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016149.296670:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.296681:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016149.296687:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.296692:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366ac0/0xa50fd972cb8e7903 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.296730:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334041088:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016149.305916:0:4749:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1734016149.305923:0:4749:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.305930:0:4749:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.305934:0:4749:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016149.305954:0:4749:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016149.305965:0:4749:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016149.306010:0:4749:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 2/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.306016:0:4749:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016149.306034:0:4749:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 1/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.306040:0:4749:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c0b40/0xa50fd972cb8e78ee lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x5e3a3adc27117d46 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.306060:0:4749:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016149.306063:0:4749:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.306068:0:4749:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.306071:0:4749:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016149.306079:0:4749:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016149.311667:0:4749:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016149.311720:0:4749:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.311728:0:4749:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016149.313353:0:4749:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.313362:0:4749:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e78e7 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4395008->4403199) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117d3f expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.313385:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016149.313386:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016149.313389:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016149.315421:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334041216:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016149.315448:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5509120->5521407] (req 5509120->5521407) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.315457:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5509120->5521407] (req 5509120->5521407) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.316149:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5509120->5521407] (req 5509120->5521407) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.325811:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335459328 00000100:00100000:3.0:1734016149.325821:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335459328 00000100:00100000:3.0:1734016149.325857:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12965 00000100:00100000:3.0:1734016149.325863:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:9876:x1818247335459328:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016149.325876:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5509120->5521407] (req 5509120->5521407) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.325890:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [5509120->5521407] (req 5509120->5521407) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.325897:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.325905:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.325912:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.325920:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.325944:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa365d40) 00000100:00100000:3.0:1734016149.325987:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:9876:x1818247335459328:12345-192.168.201.116@tcp:105: Request processed in 122us (1066us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.325992:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12965 00010000:00010000:0.0:1734016149.326156:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.326165:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.326184:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.326198:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 6088193, setting rss=6088193; kms=6088193, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.326207:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.326212:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016149.326233:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 264 00000020:00010000:0.0:1734016149.347873:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016149.347877:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.347884:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811930 (ffff88012dc12128/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016149.347898:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.347913:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016149.347926:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.347928:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811930 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016149.347943:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016149.348010:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016149.348015:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016149.348034:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016149.348049:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.348058:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016149.348067:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016149.348069:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e77f9 00000080:00010000:0.0:1734016149.348073:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e77f9 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016149.348076:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.348080:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016149.348096:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016149.348108:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016149.348128:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.348131:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551b0, osc lock ffff8800a9f22130, flags 0x400001002 00000008:00010000:0.0:1734016149.348136:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.348138:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a551f0, osc lock ffff8800a9f22e40, flags 0x400001002 00010000:00010000:0.0:1734016149.348159:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.348168:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.348223:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.348235:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.348243:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.348249:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.348280:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334041728:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.354924:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016149.354942:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335459456 00000100:00100000:3.0:1734016149.354950:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335459456 00000100:00100000:3.0:1734016149.354986:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12966 00000100:00100000:3.0:1734016149.354993:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f858c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335459456:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016149.355017:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.355027:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016149.355051:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016149.355054:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 6088193 mtime: 1734016148 atime: 1734016149 ctime: 1734016149 blocks: 264 00000020:00010000:3.0:1734016149.355057:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016149.355058:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 6088193 mtime: 1734016148 atime: 1734016149 ctime: 1734016149 blocks: 264 00000100:00100000:3.0:1734016149.355101:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f858c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335459456:12345-192.168.201.116@tcp:106: Request processed in 107us (1291us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.355107:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12966 00010000:00010000:0.0:1734016149.379375:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.379397:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.379411:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.379418:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.379426:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.379446:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6088193, setting rss=6088193; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016149.379499:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.379506:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.379514:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.379520:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.379525:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.379536:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016149.379542:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.379549:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1200/0xa50fd972cb8e7911 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.379611:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334041728:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016149.395762:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 288 00000020:00010000:0.0:1734016149.395769:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016149.395771:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.395778:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7cf0 (ffff88012dc122d8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.395782:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7d30 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.395785:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.395787:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7cf0 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.395790:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7d30 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016149.395803:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016149.395816:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10282497 00000080:00200008:0.0:1734016149.395896:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 4891078, type: SEEK_SET 00000080:00200008:0.0:1734016149.395902:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 4891078, type: SEEK_SET, rc = 4891078 00000080:00200008:0.0:1734016149.395911:0:10406:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 4891078, count: 4670 00000080:00200000:0.0:1734016149.395917:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016149.395932:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016149.395963:0:10406:0:(rw.c:1991:ll_readpage()) fast read pgno: 1194 00000080:00200000:0.0:1734016149.395970:0:10406:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 4891078, bytes: 4670 00000080:00200000:0.0:1734016149.395974:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:0.0:1734016149.395980:0:10406:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b3a5cd48[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:0.0:1734016149.395981:0:10406:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [4891078, 4895748) 0 0 00000080:00200000:0.0:1734016149.395983:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1734016149.396006:0:10406:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [4891078, 4895748) 4895748, 4670 00020000:00200000:0.0:1734016149.396010:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1734016149.396026:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [1194, 1195] 00000008:00010000:0.0:1734016149.396040:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1734016149.396043:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369991b0, osc lock ffff8800a9f22850, flags 0x0 00010000:00010000:0.0:1734016149.396066:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.396073:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [696320->704511] (req 696320->704511) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.396103:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [696320->704511] (req 696320->704511) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.396121:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2a00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334041984:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016149.396175:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55300 going to sleep for 11 seconds 00000100:00100000:3.0:1734016149.398733:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335459712 00000100:00100000:3.0:1734016149.398743:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335459712 00000100:00100000:3.0:1734016149.398785:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12968 00000100:00100000:3.0:1734016149.398793:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335459712:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016149.398809:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c06c0/0xa50fd972cb8e78a8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4063232->4067327) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117d00 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.399106:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016149.399113:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335459712:12345-192.168.201.116@tcp:104: Request processed in 321us (492us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.399121:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12968 00000100:00100000:0.0:1734016149.409753:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2a00 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334041984:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016149.409796:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [696320->704511] (req 696320->704511) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.409811:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [696320->704511] (req 696320->704511) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.409820:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [696320->704511] (req 696320->704511) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.410584:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335459840 00000100:00100000:3.0:1734016149.410592:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335459840 00000100:00100000:3.0:1734016149.411298:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12969 00000100:00100000:3.0:1734016149.411305:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:9876:x1818247335459840:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016149.411320:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [696320->704511] (req 696320->704511) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.411329:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [696320->704511] (req 696320->704511) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.411337:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.411346:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.411352:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.411361:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.411992:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c1200) 00010000:00010000:0.0:1734016149.412212:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.412223:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.412249:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.412262:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 3359509, setting rss=3359509; kms=3359509, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.412272:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.412279:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016149.412293:0:10406:0:(vvp_io.c:840:vvp_io_read_start()) f16b.sanityn: read [4891078, 4895748) 00000080:00200000:0.0:1734016149.412319:0:10406:0:(vvp_io.c:899:vvp_io_read_start()) count:4670 iocb pos:4891078 00010000:00010000:0.0:1734016149.412375:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.412387:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.412394:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 5/1,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.412399:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 5/1,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.412404:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.417646:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.417671:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.417718:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.417734:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.418164:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:9876:x1818247335459840:12345-192.168.201.116@tcp:105: Request processed in 6850us (7781us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.418175:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12969 00010000:00010000:0.0:1734016149.421537:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.421555:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.512347:0:1982:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a55300 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.513021:0:1982:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55300 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c1200/0xa50fd972cb8e791f lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 696320->704511) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d77 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.513053:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334042368:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016149.517606:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335459968 00000100:00100000:3.0:1734016149.517614:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335459968 00000100:00100000:3.0:1734016149.517649:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12970 00000100:00100000:3.0:1734016149.517656:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335459968:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016149.517667:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.517675:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016149.517698:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1846, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016149.517700:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 6088193 mtime: 1734016148 atime: 1734016149 ctime: 1734016148 blocks: 264 00000020:00010000:3.0:1734016149.517703:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016149.517704:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 6088193 mtime: 1734016148 atime: 1734016149 ctime: 1734016148 blocks: 264 00000100:00100000:3.0:1734016149.517750:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335459968:12345-192.168.201.116@tcp:106: Request processed in 94us (523us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.517757:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12970 00000100:00100000:3.0:1734016149.532603:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335460096 00000100:00100000:3.0:1734016149.532612:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335460096 00000100:00100000:3.0:1734016149.532661:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12971 00000100:00100000:3.0:1734016149.532669:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335460096:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016149.532684:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.532698:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016149.532723:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1846, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016149.532724:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 6088193 mtime: 1734016148 atime: 1734016149 ctime: 1734016148 blocks: 264 00000020:00010000:3.0:1734016149.532728:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 3, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016149.532729:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 6088193 mtime: 1734016148 atime: 1734016149 ctime: 1734016148 blocks: 264 00000100:00100000:3.0:1734016149.532776:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335460096:12345-192.168.201.116@tcp:106: Request processed in 107us (832us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.532782:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12971 00010000:00010000:3.0:1734016149.541717:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.541738:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.541753:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.541760:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.541768:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016149.541781:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6088193, setting rss=6088193; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016149.541795:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.541802:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.541809:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.541817:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016149.541822:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.541834:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016149.541840:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.541846:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7934 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.542544:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334043008:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.552519:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016149.552540:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335460224 00000100:00100000:3.0:1734016149.552548:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335460224 00000100:00100000:3.0:1734016149.552585:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335460352 00000100:00100000:3.0:1734016149.552587:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335460352 00000100:00100000:3.0:1734016149.552602:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12972 00000100:00100000:3.0:1734016149.552609:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335460224:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016149.552627:0:4742:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781c6c0/0xa50fd972cb8e792d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117d7e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.553985:0:4742:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016149.554007:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a3480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335460224:12345-192.168.201.116@tcp:104: Request processed in 1384us (1695us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.554015:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12972 00010000:00010000:0.0:1734016149.557585:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016149.557620:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334043264:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016149.562747:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334043136:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016149.562773:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3829760->3837951] (req 3829760->3837951) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.562783:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3829760->3837951] (req 3829760->3837951) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.562790:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3829760->3837951] (req 3829760->3837951) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.575507:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335460480 00000100:00100000:3.0:1734016149.575516:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335460480 00000100:00100000:3.0:1734016149.575566:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12974 00000100:00100000:3.0:1734016149.575572:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335460480:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016149.575588:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3829760->3837951] (req 3829760->3837951) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.575599:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [3829760->3837951] (req 3829760->3837951) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.575607:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.575618:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.575626:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.575634:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.575660:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa366880) 00000100:00100000:3.0:1734016149.575709:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335460480:12345-192.168.201.116@tcp:105: Request processed in 137us (356us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.575716:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12974 00010000:00010000:0.0:1734016149.575894:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.575904:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.575931:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.575944:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 3359509, setting rss=3359509; kms=3359509, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.575956:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.575962:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016149.575988:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 288 00000100:00100000:0.0:1734016149.576073:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016149.576088:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334043520:192.168.201.116@tcp:22:fsx.0 00000100:00100000:0.0:1734016149.581517:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334043520:192.168.201.116@tcp:22:fsx.0 00000100:00100000:3.0:1734016149.582025:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334043648:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.582097:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016149.582103:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334043776:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.583827:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016149.586055:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016149.594251:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594275:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594291:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594297:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594305:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016149.594327:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3359509, setting rss=3359509; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016149.594335:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594341:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594348:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594354:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016149.594358:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594369:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016149.594375:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594381:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa364b40/0xa50fd972cb8e7949 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.594427:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334043776:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016149.594447:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594454:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594481:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594488:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594493:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016149.594498:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 6088193, setting rss=6088193; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016149.594505:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594511:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594516:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594520:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367840 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016149.594523:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594529:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016149.594532:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.594542:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa367840/0xa50fd972cb8e7942 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.594555:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334043648:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.594991:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334044032:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.595063:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016149.595069:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334043904:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.601059:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016149.601130:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:3.0:1734016149.612165:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.612188:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.612200:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.612206:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.612211:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016149.612232:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3359509, setting rss=3359509; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016149.612244:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.612250:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.612260:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.612266:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016149.612271:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.612284:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016149.612289:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.612295:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365200/0xa50fd972cb8e7957 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.612326:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334044032:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016149.613971:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7600 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.614008:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2723840 2727935) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016149.614040:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981880 - object 0x280000400:8 offset 2723840<>2727936 00000100:00100000:3.0:1734016149.614056:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016149.614067:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334044288:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016149.617119:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7600 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016149.617162:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7600 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.617189:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334044288:192.168.201.116@tcp:3:fsx.0 00000080:00200000:0.0:1734016149.668710:0:10406:0:(rw26.c:1006:ll_write_end()) pos 6922240, len 3755, copied 3755 00000080:00200000:0.0:1734016149.668720:0:10406:0:(rw26.c:1044:ll_write_end()) page@ffff8800a8ba4798[3 ffff8800b3a5cdc8 2 1 ffff88012dbed2a0] 00000080:00200000:0.0:1734016149.668722:0:10406:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a9f600 1fffff0000082d 2:0 ffff8800a8ba4798 1690 lru 00000080:00200000:0.0:1734016149.668730:0:10406:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a8ba47f8 666: 1< 1 - - > 2< 2727936 0 4096 0x0 0x40009 | ffff8800aa960708 ffff8800a90061a0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1734016149.668732:0:10406:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a8ba4798 00000080:00200000:0.0:1734016149.668733:0:10406:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:0.0:1734016149.668740:0:10406:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 2, from 2791, to 3755 00000008:00010000:0.0:1734016149.668773:0:10406:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800a8a55300 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.668799:0:10406:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800a8a55300 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016149.668811:0:10406:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af134248, count 2, dirtied 2 00000080:00200000:0.0:1734016149.668815:0:10406:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 2 pages 5060 bytes, tot: 5060 00000080:00200000:0.0:1734016149.668818:0:10406:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 5060 00000020:00010000:0.0:1734016149.668835:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016149.668838:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.668841:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811230 (ffff88012dc122d8/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016149.668848:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.668858:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016149.668875:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.668877:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811230 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016149.668884:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 5060 bytes 00000080:00200000:0.0:1734016149.668890:0:10406:0:(file.c:1930:ll_file_io_generic()) Range unlock [1689, 1690] 00000080:00200000:0.0:1734016149.668900:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016149.668904:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 5060, restart: 0 00000080:00200000:0.0:1734016149.668918:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 288 00000080:00200000:0.0:1734016149.668924:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 5060 00000080:00200008:0.0:1734016149.668945:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016149.668951:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016149.668970:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016149.668985:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.669006:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016149.669017:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016149.669020:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e7800 00000080:00010000:0.0:1734016149.669025:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7800 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016149.669028:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.669034:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016149.669050:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016149.669060:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016149.669077:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.669080:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a559f0, osc lock ffff8800a9f22be0, flags 0x400001002 00000008:00010000:0.0:1734016149.669085:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.669086:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55a30, osc lock ffff8800a9f22000, flags 0x400001002 00010000:00010000:0.0:1734016149.669100:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669107:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669116:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669122:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669135:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669141:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669147:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669152:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016149.669160:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 288 00000020:00010000:0.0:1734016149.669163:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016149.669165:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.669167:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a559f0 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.669169:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55a30 (ffff88012dc122d8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.669171:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.669173:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a559f0 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.669176:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55a30 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016149.669181:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016149.669192:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016149.669196:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016149.669199:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016149.669203:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.669205:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a559f0, osc lock ffff8800a9f22000, flags 0x400001002 00000008:00010000:0.0:1734016149.669208:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.669209:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55a30, osc lock ffff8800a9f22be0, flags 0x400001002 00010000:00010000:0.0:1734016149.669214:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669221:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669228:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669234:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669243:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669250:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669256:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.669262:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016149.669271:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10282497 i_blocks 288 00000020:00010000:0.0:1734016149.669274:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016149.669276:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.669278:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a559f0 (ffff88012dc122d8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.669281:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55a30 (ffff88012dc12128/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.669282:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbed240 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.669284:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a559f0 (ffff88012dc12488/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.669287:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55a30 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016149.669291:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016149.669297:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 10282497 00000080:00200008:0.0:1734016149.669347:0:10406:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8) current size 10282497, valid attrs 2068, mode 8801, uid -1227447776, gid -30720, new size 9283538, atime 1734016149.000000000, mtime 1734016149.000000000, ctime 1734016149.000000000 00010000:00010000:0.0:1734016149.669366:0:10406:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.669380:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.669420:0:10406:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.669426:0:10406:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016149.669429:0:10406:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364fc0/0xa50fd972cb8e7800 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27117c58 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000020:00100000:0.0:1734016149.670780:0:10406:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff88012a571000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1734016149.670794:0:10406:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a78e2300 x1818247334044800/t0(0) o36->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016149.677203:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016149.678840:0:4749:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016149.678851:0:4749:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.678860:0:4749:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa364900 already unused, calling callback (ffffffffa10571e0) 00010000:00010000:0.0:1734016149.678866:0:4749:0:(ldlm_request.c:1579:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.678874:0:4749:0:(ldlm_request.c:1589:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.678882:0:4749:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.678908:0:4749:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800a781c900/0xa50fd972cb8e5f33 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115fb0 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.678952:0:4749:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 2/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.678958:0:4749:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016149.680302:0:4749:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.680313:0:4749:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa364900/0xa50fd972cb8e77f9 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117c4a expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.680355:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016149.680357:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016149.680359:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016149.680384:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016149.680409:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334044928:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016149.686277:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334044928:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016149.699195:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334045312:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016149.699226:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.699238:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.699250:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.699260:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.699270:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.699277:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.699283:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016149.699294:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78a2680 x1818247334045312/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016165 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016149.699305:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016149.699322:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.699335:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016149.699391:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016149.699396:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e795e 00000080:00010000:0.0:1734016149.699402:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016149.699404:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e795e from it ffff8801369fbcd8 00010000:00010000:0.0:1734016149.699407:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.699414:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016149.699447:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016149.699456:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000100:00100000:0.0:1734016149.713155:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334045568:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016149.713184:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.713196:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.713211:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.713221:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.713231:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.713239:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016149.713245:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016149.713256:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78a3480 x1818247334045568/t0(0) o101->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016165 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016149.713266:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016149.713277:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.713290:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b5861000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016149.713352:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016149.713358:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e7965 00000080:00010000:0.0:1734016149.713363:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016149.713366:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7965 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016149.713369:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.713376:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016149.713417:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016149.713428:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016149.713454:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.713457:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a559f0, osc lock ffff8800a9f22be0, flags 0x400001002 00000008:00010000:0.0:1734016149.713570:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.713573:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55a30, osc lock ffff8800a9f22000, flags 0x400001002 00010000:00010000:0.0:1734016149.713598:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.713607:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.714353:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.714360:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.718636:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016149.718656:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335461248 00000100:00100000:3.0:1734016149.718666:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335461248 00000100:00100000:3.0:1734016149.718720:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12980 00000100:00100000:3.0:1734016149.718729:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2de450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335461248:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016149.718745:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.718756:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016149.718787:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1846, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016149.718789:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5089234 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 232 00000020:00010000:3.0:1734016149.718793:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016149.718794:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5089234 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 232 00000100:00100000:3.0:1734016149.718843:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2de450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335461248:12345-192.168.201.116@tcp:106: Request processed in 115us (449us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.718849:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12980 00010000:00010000:0.0:1734016149.722824:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.722844:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.722857:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.722863:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.722870:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.722886:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016149.722894:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.722899:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.722906:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.722912:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c0b40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.722917:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.722926:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016149.722931:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.722936:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c0b40/0xa50fd972cb8e7973 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.723019:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334045824:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016149.723039:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.723046:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.723052:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.723057:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.723062:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.723068:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5089234, setting rss=5089234; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016149.723087:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.723092:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.723097:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.723101:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c1d40 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.723103:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.723110:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016149.723112:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.723117:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a78c1d40/0xa50fd972cb8e796c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.723132:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334045696:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016149.723560:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334046080:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016149.723620:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016149.723625:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334045952:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.728093:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016149.728366:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016149.728391:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335461504 00000100:00100000:3.0:1734016149.728400:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335461504 00000100:00100000:3.0:1734016149.728560:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335461632 00000100:00100000:3.0:1734016149.728562:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335461632 00010000:00010000:0.0:1734016149.733513:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733534:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733550:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733557:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733563:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.733580:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5089234, setting rss=5089234; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016149.733588:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733594:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733600:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733607:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781e1c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.733614:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733624:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016149.733629:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733635:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e797a lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.733685:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334045952:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016149.733704:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733719:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733726:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733730:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733735:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.733741:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016149.733770:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733777:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733782:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733786:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781d680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.733788:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733797:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016149.733800:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.733805:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e7981 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.733820:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334046080:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016149.748363:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016149.748374:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334046976:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016149.748417:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334047104:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.751561:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016149.752147:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335461760 00000100:00100000:3.0:1734016149.752154:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335461760 00000100:00100000:3.0:1734016149.752281:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335461888 00000100:00100000:3.0:1734016149.752283:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335461888 00000100:00100000:3.0:1734016149.752296:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12983 00000100:00100000:3.0:1734016149.752303:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335461760:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016149.752319:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.752328:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016149.752353:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1846, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016149.752355:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5089234 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 232 00000020:00010000:3.0:1734016149.752358:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016149.752359:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 5089234 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 232 00000100:00100000:3.0:1734016149.752660:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7683:x1818247335461760:12345-192.168.201.116@tcp:106: Request processed in 357us (1110us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.752666:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12983 00010000:00010000:0.0:1734016149.755087:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755105:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755119:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755126:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755132:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.755148:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016149.755156:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755162:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755169:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755176:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781d680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.755181:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755191:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016149.755196:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755202:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a781d680/0xa50fd972cb8e799d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.755249:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334047104:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016149.755270:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755282:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755289:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755293:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755299:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.755306:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5089234, setting rss=5089234; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016149.755666:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755673:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755678:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755682:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781c480 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.755684:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755695:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016149.755698:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.755702:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781c480/0xa50fd972cb8e7996 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.755717:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334046976:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016149.756146:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334047360:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016149.756212:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016149.756218:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334047232:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.759774:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335462016 00000100:00100000:3.0:1734016149.759786:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335462016 00000100:00100000:3.0:1734016149.759823:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12985 00000100:00100000:3.0:1734016149.759832:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335462016:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016149.759848:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.759859:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016149.759886:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1846, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016149.759888:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 40 00000020:00010000:3.0:1734016149.759891:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016149.759892:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 40 00000100:00100000:3.0:1734016149.759936:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7683:x1818247335462016:12345-192.168.201.116@tcp:106: Request processed in 106us (590us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.759941:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12985 00000100:00100000:3.0:1734016149.759976:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016149.760004:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335462144 00000100:00100000:3.0:1734016149.760007:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335462144 00000100:00100000:3.0:1734016149.760023:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12986 00000100:00100000:3.0:1734016149.760027:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7981180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335462144:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016149.760033:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.760038:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa365d40/0xa50fd972cb8e790a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5509120->5521407) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117d62 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016149.760048:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 1846, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016149.760049:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5089234 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 232 00000020:00010000:3.0:1734016149.760052:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f23e8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016149.760053:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 5089234 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 232 00000100:00100000:3.0:1734016149.760067:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7981180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335462144:12345-192.168.201.116@tcp:106: Request processed in 40us (91us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.760071:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12986 00010000:00010000:0.0:1734016149.763352:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.763372:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.763384:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.763389:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.763395:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.763411:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5089234, setting rss=5089234; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016149.763419:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.763424:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.763430:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.763436:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a781e1c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.763440:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.763450:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016149.763455:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.763460:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800a781e1c0/0xa50fd972cb8e79a4 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.763588:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334047232:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.767443:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000008:00010000:3.0:1734016149.783691:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7e40 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.783762:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3870720 3874815) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016149.783929:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7981880 - object 0x240000400:6 offset 0<>4194304 00000100:00100000:3.0:1734016149.783950:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016149.783963:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334047872:192.168.201.116@tcp:3:fsx.0 00000020:00010000:0.0:1734016149.843262:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_release():233 00000020:00010000:0.0:1734016149.843268:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.843272:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b3811f30 (ffff88012dc12638/1) at cl_lock_cancel():150 00010000:00010000:0.0:1734016149.843279:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.843289:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:0.0:1734016149.843303:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a7a34128 (ffff88012dc12488/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.843304:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b3811f30 (ffff88012dc12488/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016149.843312:0:10406:0:(vvp_io.c:533:vvp_io_advance()) advancing 4375 bytes 00000080:00200000:0.0:1734016149.843321:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1734016149.843324:0:10406:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 4375, restart: 0 00000080:00200000:0.0:1734016149.843328:0:10406:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 4375 00000080:00200008:0.0:1734016149.843349:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1348c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016149.843354:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1348c8),name=f16b.sanityn 00000002:00010000:0.0:1734016149.843368:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016149.843379:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.843388:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016149.843398:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) 00000080:00010000:0.0:1734016149.843400:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1348c8) for lock 0xa50fd972cb8e7965 00000080:00010000:0.0:1734016149.843404:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e7965 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016149.843407:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.843413:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a5ff800 lock: ffff8800aa366d00/0xa50fd972cb8e7965 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117dd2 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016149.843428:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016149.843439:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016149.843456:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.843459:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55e70, osc lock ffff8800a9f23430, flags 0x400001002 00000008:00010000:0.0:1734016149.843514:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.843516:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55eb0, osc lock ffff8800a9f22d10, flags 0x400001002 00010000:00010000:0.0:1734016149.843526:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.843534:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.843543:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.843549:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.843568:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.843575:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.843636:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016149.843652:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334048000:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.851824:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016149.857970:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.858108:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.858122:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.858128:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.858135:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.858151:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016149.858949:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.858958:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.858964:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.858970:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c21c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.858976:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.859183:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016149.859189:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.859196:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c21c0/0xa50fd972cb8e79b9 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.860105:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334048000:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.860398:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016149.860410:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334048128:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.864218:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000080:00200000:0.0:1734016149.870141:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9283538 i_blocks 272 00000020:00010000:0.0:1734016149.870147:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_release():233 00000020:00010000:0.0:1734016149.870150:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.870155:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7e70 (ffff8800a8a54bd8/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.870159:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ee7eb0 (ffff88012dc12878/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016149.870162:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff88012dbedff0 (ffff88012dc12638/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.870163:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7e70 (ffff88012dc12638/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016149.870167:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ee7eb0 (ffff88012dc12638/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016149.870178:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016149.870189:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9283538 00000080:00a00000:0.0:1734016149.870252:0:10406:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800b6e4e948 start=0x7f17cf5c9000 end=0x7f17cf5cc000 vm_flags=0xf9 00000080:00200000:0.0:1734016149.870261:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016149.870270:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016149.870290:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.870292:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f30, osc lock ffff8800a9f23300, flags 0x400001002 00000008:00010000:0.0:1734016149.870298:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016149.870299:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f70, osc lock ffff8800a9f224c0, flags 0x400001002 00010000:00010000:0.0:1734016149.870313:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.870323:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.870331:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.870337:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.870363:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e79c7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.870369:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c1b00/0xa50fd972cb8e79c7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.875884:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016149.888033:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016149.888716:0:4749:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016149.888723:0:4749:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.888730:0:4749:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016149.888734:0:4749:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.888773:0:4749:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55300 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.888789:0:4749:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800a8a55300 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.888826:0:4749:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2723840 2727935) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x800429400020000 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016149.888889:0:4749:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e3b80 - object 0x280000400:8 offset 2723840<>2732032 00000080:00200000:0.0:1734016149.898527:0:4749:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1734016149.902077:0:4749:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1734016149.902139:0:4749:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 2/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.902148:0:4749:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016149.902187:0:4749:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 1/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.902193:0:4749:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366880/0xa50fd972cb8e793b lrc: 0/0,0 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3829760->3837951) gid 0 flags: 0x804c69400020020 nid: local remote: 0x5e3a3adc27117d93 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.902212:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016149.902214:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016149.902217:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016149.902242:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016149.902264:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334048640:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016149.915104:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00010000:00010000:0.0:1734016149.915652:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.915663:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.915685:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.915695:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.915704:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.915710:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.915770:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.915781:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.928282:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.928297:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.928666:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:0.0:1734016149.928683:0:1981:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55a80 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016149.928738:0:1981:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (282624 286719) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016149.928867:0:1981:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a78e0700 - object 0x280000400:8 offset 0<>4194304 00000100:00100000:0.0:1734016149.928891:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334048896:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016149.978020:0:1981:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a55a80 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016149.978604:0:1981:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55a80 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016149.978627:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334048896:192.168.201.116@tcp:3:fsx.0 00010000:00010000:3.0:1734016149.985163:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e79d5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985184:0:1984:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e79d5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117e65 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985198:0:1984:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e79d5 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117e65 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985209:0:1984:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e79d5 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117e65 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016149.985222:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 5089234, setting rss=5089234; kms=5089234, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e79d5 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117e65 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985233:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e79d5 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117e65 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985240:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e79d5 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117e65 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985245:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e79d5 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117e65 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985547:0:1984:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e79d5 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117e65 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.985567:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334049024:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016149.985594:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e79dc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985603:0:1984:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e79dc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117e6c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985609:0:1984:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e79dc lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117e6c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985616:0:1984:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e79dc lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117e6c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016149.985624:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e79dc lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117e6c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985822:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e79dc lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117e6c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985829:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e79dc lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117e6c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985839:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e79dc lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x5e3a3adc27117e6c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016149.985845:0:1984:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e79dc lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x5e3a3adc27117e6c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016149.985855:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334049152:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016149.995354:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335463168 00000100:00100000:3.0:1734016149.995362:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335463168 00000100:00100000:3.0:1734016149.995395:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12994 00000100:00100000:3.0:1734016149.995402:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:8c598caf-7141-4c19-8566-0b7731516db9+5:7704:x1818247335463168:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016149.995415:0:4741:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1734016149.995507:0:4741:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016149.995514:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:8c598caf-7141-4c19-8566-0b7731516db9+5:7704:x1818247335463168:12345-192.168.201.116@tcp:104: Request processed in 112us (420us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016149.995521:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12994 00010000:00010000:0.0:1734016149.995892:0:4750:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016149.995898:0:4750:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.995905:0:4750:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365200 already unused, calling callback (ffffffffa10571e0) 00010000:00010000:0.0:1734016149.995908:0:4750:0:(ldlm_request.c:1579:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.995915:0:4750:0:(ldlm_request.c:1589:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.995923:0:4750:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.995943:0:4750:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.995976:0:4750:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 2/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.995981:0:4750:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1734016149.996183:0:4750:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016149.996190:0:4750:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e795e lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x5e3a3adc27117dc4 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000100:00100000:3.0:1734016150.001260:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0000 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334049408:192.168.201.116@tcp:36:fsx.0 00000080:00200000:3.0:1734016150.001310:0:10406:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:3.0:1734016150.001316:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:3.0:1734016150.001333:0:10406:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016150.001355:0:10406:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:3.0:1734016150.001369:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016150.001372:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d4570, osc lock ffff8800a8a26850, flags 0x0 00000008:00010000:3.0:1734016150.001375:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1734016150.001377:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8801369d45b0, osc lock ffff8800a8a265f0, flags 0x0 00010000:00010000:3.0:1734016150.001392:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800a9dd4000 type 11 mode 2 res 9663677440/6 (0 18446744073709551615) 00010000:00010000:3.0:1734016150.001408:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.001417:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.001450:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016150.002043:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334049664:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.002783:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8801369d4e40 going to sleep for 11 seconds 00000100:00100000:3.0:1734016150.005006:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335463296 00000100:00100000:3.0:1734016150.005015:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335463296 00000100:00100000:3.0:1734016150.005053:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335463424 00000100:00100000:3.0:1734016150.005055:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335463424 00000100:00100000:3.0:1734016150.005070:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12995 00000100:00100000:3.0:1734016150.005077:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335463296:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016150.005092:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e79d5 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117e65 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.005169:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1734016150.005175:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335463296:12345-192.168.201.116@tcp:104: Request processed in 98us (395us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.005182:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12995 00000100:00100000:3.0:1734016150.005198:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12996 00000100:00100000:3.0:1734016150.005202:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335463424:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016150.005209:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e79b2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3870720->3878911) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117e42 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.005797:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016150.005803:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335463424:12345-192.168.201.116@tcp:104: Request processed in 599us (935us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.005809:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12996 00010000:00010000:0.0:1734016150.007320:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016150.007344:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016150.007361:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334049792:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016150.013514:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334049664:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016150.013544:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.013554:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.013561:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016150.014537:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1734016150.014553:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334049920:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1734016150.015193:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334049792:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016150.015823:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335463552 00000100:00100000:3.0:1734016150.015829:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335463552 00000100:00100000:3.0:1734016150.015871:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12997 00000100:00100000:3.0:1734016150.015877:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335463552:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016150.015888:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.015897:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.015905:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.015913:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.015922:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.015931:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.015958:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa348fc0) 00000100:00100000:3.0:1734016150.016003:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335463552:12345-192.168.201.116@tcp:105: Request processed in 124us (248us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.016009:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12997 00010000:00010000:0.0:1734016150.016435:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.016444:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.016462:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016150.016501:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 5089234, setting rss=5089234; kms=5089234, end=18446744073709551615 ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.016510:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.016517:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.016537:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800a9dd5400 type 11 mode 2 res 10737419264/8 (0 18446744073709551615) 00010000:00010000:0.0:1734016150.016547:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.016552:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.016577:0:10406:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 3/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016150.016594:0:10406:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334050048:192.168.201.116@tcp:101:fsx.0 00000100:00100000:0.0:1734016150.016634:0:10406:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800a8a55a80 going to sleep for 11 seconds 00000100:00100000:0.0:1734016150.017099:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334049920:192.168.201.116@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1734016150.018308:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335463680 00000100:00100000:3.0:1734016150.018316:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335463680 00000100:00100000:3.0:1734016150.018355:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335463808 00000100:00100000:3.0:1734016150.018358:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335463808 00000100:00100000:3.0:1734016150.018372:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 12998 00000100:00100000:3.0:1734016150.018377:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335463680:12345-192.168.201.116@tcp:104: 00010000:00010000:3.0:1734016150.018389:0:9558:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa366f40/0xa50fd972cb8e79dc lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x5e3a3adc27117e6c expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.018755:0:9558:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1734016150.018762:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335463680:12345-192.168.201.116@tcp:104: Request processed in 384us (726us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.018769:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 12998 00010000:00010000:0.0:1734016150.018906:0:4749:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1734016150.018910:0:4749:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x800420000020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.018916:0:4749:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa365680 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016150.018919:0:4749:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa365680/0xa50fd972cb8e79ce lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 282624->286719) gid 0 flags: 0x800428400020000 nid: local remote: 0x5e3a3adc27117e5e expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016150.018929:0:4749:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000100:00100000:3.0:1734016150.020173:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job fsx:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:10406:1818247334050048:192.168.201.116@tcp:101:fsx.0 00010000:00010000:3.0:1734016150.020192:0:10406:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.020200:0:10406:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.020208:0:10406:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 4/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016150.033366:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016150.033965:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335463936 00000100:00100000:3.0:1734016150.033974:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335463936 00000100:00100000:3.0:1734016150.035309:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13000 00000100:00100000:3.0:1734016150.035316:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012f85b450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:9876:x1818247335463936:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016150.035328:0:4742:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.035338:0:4742:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.035346:0:4742:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.035360:0:4742:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.035367:0:4742:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.035375:0:4742:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.036764:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.036774:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.036798:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016150.036809:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.036818:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.036885:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.037004:0:4742:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a78c3a80) 00000100:00100000:3.0:1734016150.037407:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012f85b450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:9876:x1818247335463936:12345-192.168.201.116@tcp:105: Request processed in 2087us (6095us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.037414:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13000 00000100:00100000:0.0:1734016150.038300:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016150.038315:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334050432:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016150.038388:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334050560:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016150.042918:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78e0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334050432:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016150.044080:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334050560:192.168.201.116@tcp:10:fsx.0 00000100:00100000:0.0:1734016150.048884:0:10406:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job fsx:8c598caf-7141-4c19-8566-0b7731516db9:10406:1818247334050688:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016150.048906:0:10406:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016150.048916:0:10406:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016150.048926:0:10406:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016150.048934:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016150.048941:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016150.048947:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016150.048954:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1734016150.048961:0:10406:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a78a2680 x1818247334050688/t0(0) o101->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016166 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:0.0:1734016150.048969:0:10406:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1734016150.048978:0:10406:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016150.049002:0:10406:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6217000 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:0.0:1734016150.049066:0:10406:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1734016150.049071:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e79f1 00000080:00010000:0.0:1734016150.049075:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016150.049078:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e79f1 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016150.049080:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016150.049086:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016150.049109:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016150.049116:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016150.049136:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016150.049139:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f30, osc lock ffff8800a9f22ab0, flags 0x400001002 00000008:00010000:0.0:1734016150.049152:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016150.049153:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f70, osc lock ffff8800a9f22f70, flags 0x400001002 00010000:00010000:0.0:1734016150.049172:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.049179:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016150.049493:0:1982:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1734016150.049505:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334050816:192.168.201.116@tcp:101:fsx.0 00010000:00010000:0.0:1734016150.049610:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e79ff lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.049617:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c3180/0xa50fd972cb8e79ff lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016150.049642:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334050944:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.053980:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335464064 00000100:00100000:3.0:1734016150.053996:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335464064 00000100:00100000:3.0:1734016150.054035:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13001 00000100:00100000:3.0:1734016150.054043:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335464064:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.054060:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.054072:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.054100:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.054102:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 232 00000020:00010000:3.0:1734016150.054105:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016150.054106:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 232 00000100:00100000:3.0:1734016150.054152:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335464064:12345-192.168.201.116@tcp:106: Request processed in 110us (379us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.054158:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13001 00000100:00100000:3.0:1734016150.055981:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335464192 00000100:00100000:3.0:1734016150.056702:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335464192 00000100:00100000:3.0:1734016150.056749:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13002 00000100:00100000:3.0:1734016150.056756:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335464192:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.056773:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.056784:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.056810:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.056812:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 32 00000020:00010000:3.0:1734016150.056816:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016150.056817:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 32 00000100:00100000:3.0:1734016150.056864:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335464192:12345-192.168.201.116@tcp:106: Request processed in 109us (1635us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.056870:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13002 00010000:00010000:0.0:1734016150.058004:0:1982:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.058018:0:1982:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.058030:0:1982:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.058036:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.058046:0:1982:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016150.058059:0:1982:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016150.058066:0:1982:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.058072:0:1982:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.058078:0:1982:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.058085:0:1982:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c2400 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016150.058090:0:1982:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.058126:0:1982:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016150.058132:0:1982:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.058138:0:1982:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2400/0xa50fd972cb8e79f8 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016150.058935:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334050816:192.168.201.116@tcp:101:fsx.0 00000080:00200000:0.0:1734016150.060923:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7353208 i_blocks 264 00000020:00010000:0.0:1734016150.060929:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8490 (ffff8800a8a54998/0) at cl_lock_release():233 00000020:00010000:0.0:1734016150.060932:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8490 (ffff8800a8a54998/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016150.060937:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55f30 (ffff8800a8a54b48/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016150.060940:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55f70 (ffff8800a8a54c68/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016150.060942:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8490 (ffff8800a8a54998/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016150.060944:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55f30 (ffff8800a8a54998/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016150.060947:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55f70 (ffff8800a8a54998/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016150.060959:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016150.060979:0:10406:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1734016150.060993:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016150.061002:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016150.061015:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016150.061017:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f30, osc lock ffff8800a9f22f70, flags 0x400001002 00000008:00010000:0.0:1734016150.061021:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016150.061022:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f70, osc lock ffff8800a9f22ab0, flags 0x400001002 00010000:00010000:0.0:1734016150.061043:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e7a06 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.061051:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c18c0/0xa50fd972cb8e7a06 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.061105:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e7a0d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.061111:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c33c0/0xa50fd972cb8e7a0d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016150.064443:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13003 00000100:00100000:3.0:1734016150.064451:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335464320:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.064888:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.064901:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.064916:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.064918:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 32 00000020:00010000:3.0:1734016150.064921:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016150.064922:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 32 00000100:00100000:3.0:1734016150.064946:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:8216:x1818247335464320:12345-192.168.201.116@tcp:106: Request processed in 495us (1318us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.064951:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13003 00000080:00200000:0.0:1734016150.068968:0:10406:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7353208 i_blocks 264 00000020:00010000:0.0:1734016150.068975:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a79c8490 (ffff8800a8a54998/0) at cl_lock_release():233 00000020:00010000:0.0:1734016150.068978:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a79c8490 (ffff8800a8a54998/0) at cl_lock_cancel():150 00000020:00010000:0.0:1734016150.068983:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55f30 (ffff8800a8a54c68/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016150.068998:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8a55f70 (ffff8800a8a54b48/1) at cl_lock_cancel():150 00000020:00010000:0.0:1734016150.069001:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a79c8490 (ffff8800a8a54998/0) at cl_lock_fini():85 00000020:00010000:0.0:1734016150.069003:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55f30 (ffff8800a8a54998/1) at cl_lock_fini():85 00000020:00010000:0.0:1734016150.069007:0:10406:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8a55f70 (ffff8800a8a54998/1) at cl_lock_fini():85 00000080:00200000:0.0:1734016150.069022:0:10406:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1734016150.069034:0:10406:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7353208 00000080:00200008:0.0:1734016150.069054:0:10406:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af1340c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1734016150.069060:0:10406:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af1340c8),name=f16b.sanityn 00000002:00010000:0.0:1734016150.069079:0:10406:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:0.0:1734016150.069093:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016150.069104:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00010000:0.0:1734016150.069115:0:10406:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) 00000080:00010000:0.0:1734016150.069119:0:10406:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af1340c8) for lock 0xa50fd972cb8e79f1 00000080:00010000:0.0:1734016150.069125:0:10406:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0xa50fd972cb8e79f1 from it ffff8801369fbcd8 00010000:00010000:0.0:1734016150.069127:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016150.069134:0:10406:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016150.069156:0:10406:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1734016150.069166:0:10406:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:0.0:1734016150.069183:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016150.069186:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f30, osc lock ffff8800a9f22ab0, flags 0x400001002 00000008:00010000:0.0:1734016150.069190:0:10406:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1734016150.069192:0:10406:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800a8a55f70, osc lock ffff8800a9f22f70, flags 0x400001002 00010000:00010000:0.0:1734016150.069218:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.069225:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.069276:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0480/0xa50fd972cb8e7a1b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.069282:0:10406:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800a78c0480/0xa50fd972cb8e7a1b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016150.070081:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334051328:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.075245:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335464576 00000100:00100000:3.0:1734016150.075255:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335464576 00000100:00100000:3.0:1734016150.075295:0:9558:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335464704 00000100:00100000:3.0:1734016150.075296:0:9558:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335464704 00000100:00100000:3.0:1734016150.075308:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13005 00000100:00100000:3.0:1734016150.075314:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335464576:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.075330:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.075341:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.075369:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.075371:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 32 00000020:00010000:3.0:1734016150.075374:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016150.075375:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 32 00000100:00100000:3.0:1734016150.075419:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7682:x1818247335464576:12345-192.168.201.116@tcp:106: Request processed in 106us (1820us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.075425:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13005 00000100:00100000:3.0:1734016150.075438:0:9558:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13006 00000100:00100000:3.0:1734016150.075442:0:9558:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335464704:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.075448:0:9558:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.075454:0:9558:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.075480:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.075481:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 232 00000020:00010000:3.0:1734016150.075484:0:9558:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016150.075485:0:9558:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 232 00000100:00100000:3.0:1734016150.075500:0:9558:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8216:x1818247335464704:12345-192.168.201.116@tcp:106: Request processed in 57us (1166us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.075504:0:9558:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13006 00010000:00010000:0.0:1734016150.078558:0:1981:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.078566:0:1981:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.078573:0:1981:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.078583:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.078588:0:1981:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016150.078594:0:1981:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:0.0:1734016150.078603:0:1981:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.078607:0:1981:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.078612:0:1981:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.078617:0:1981:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a78c2880 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:0.0:1734016150.078619:0:1981:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.078625:0:1981:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1734016150.078627:0:1981:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.078631:0:1981:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a78c2880/0xa50fd972cb8e7a14 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016150.078651:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334051328:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.084118:0:4742:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016150.084137:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335464832 00000100:00100000:3.0:1734016150.084146:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335464832 00000100:00100000:3.0:1734016150.084189:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13007 00000100:00100000:3.0:1734016150.084196:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dd050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335464832:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.084211:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.084221:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800a78c3a80/0xa50fd972cb8e79ea lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e81 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.084250:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.084253:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 32 00000020:00010000:3.0:1734016150.084256:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5c000[0x4, 2, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016150.084257:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016149 atime: 1734016149 ctime: 1734016149 blocks: 32 00000100:00100000:3.0:1734016150.084305:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dd050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:8216:x1818247335464832:12345-192.168.201.116@tcp:106: Request processed in 109us (398us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.084311:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13007 00000100:00100000:3.0:1734016150.086866:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016150.096040:0:1976:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016150.100674:0:1975:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.116@tcp (source: 12345-192.168.201.116@tcp) 00000100:00100000:3.0:1734016150.100703:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335465216 00000100:00100000:3.0:1734016150.100712:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335465216 00000100:00100000:3.0:1734016150.101417:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13010 00000100:00100000:3.0:1734016150.101423:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7980e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335465216:12345-192.168.201.116@tcp:105: 00010000:00010000:3.0:1734016150.101432:0:4741:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [942080->954367] (req 942080->954367) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.101440:0:4741:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [942080->954367] (req 942080->954367) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.101447:0:4741:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.101455:0:4741:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.101480:0:4741:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x10080000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.101488:0:4741:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x80000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.101642:0:4741:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800aa365200) 00010000:00010000:0.0:1734016150.101737:0:10406:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.101746:0:10406:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x20000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.101771:0:10406:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x10000000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016150.101785:0:10406:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 3158904, setting rss=3158904; kms=3158904, end=18446744073709551615 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x10020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.101798:0:10406:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.101803:0:10406:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x810020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000080:00200000:0.0:1734016150.101819:0:10406:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [5137853, 5144940) 00000080:00200000:0.0:1734016150.101861:0:10406:0:(rw26.c:819:ll_write_begin()) Writing 1254 of 1469 to 2627 bytes 00000100:00100000:3.0:1734016150.101980:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7980e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:9876:x1818247335465216:12345-192.168.201.116@tcp:105: Request processed in 554us (1316us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.101995:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13010 00000100:00100000:0.0:1734016150.102715:0:1981:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:0.0:1734016150.102733:0:1981:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8a55480 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1734016150.102772:0:1981:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (942080 946175) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:0.0:1734016150.102804:0:1981:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a934dc00 - object 0x280000400:8 offset 942080<>946176 00000100:00100000:0.0:1734016150.102824:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334052352:192.168.201.116@tcp:3:fsx.0 00000008:00010000:0.0:1734016150.106694:0:1981:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8a55480 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1734016150.106727:0:1981:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8a55480 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1734016150.106744:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334052352:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016150.107076:0:1983:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee7900 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.107109:0:1983:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (950272 954367) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016150.107140:0:1983:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7983480 - object 0x280000400:8 offset 950272<>954368 00000100:00100000:3.0:1734016150.107153:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016150.107162:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334052608:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016150.109748:0:1983:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee7900 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016150.109789:0:1983:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee7900 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016150.109815:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334052608:192.168.201.116@tcp:3:fsx.0 00000100:00100000:3.0:1734016150.110656:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016150.110671:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334052736:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.116276:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335465344 00000100:00100000:3.0:1734016150.116287:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335465344 00000100:00100000:3.0:1734016150.116332:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13011 00000100:00100000:3.0:1734016150.116340:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335465344:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.116355:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.116367:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.116395:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.116398:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016150 atime: 1734016149 ctime: 1734016150 blocks: 32 00000020:00010000:3.0:1734016150.116401:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016150.116402:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016150 atime: 1734016149 ctime: 1734016150 blocks: 32 00000100:00100000:3.0:1734016150.116451:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335465344:12345-192.168.201.116@tcp:106: Request processed in 112us (297us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.116457:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13011 00010000:00010000:3.0:1734016150.120835:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.120856:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.120874:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.120881:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.120887:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016150.120900:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3158904, setting rss=3158904; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016150.120913:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.120918:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.120924:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.120931:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016150.120936:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.120945:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016150.120950:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.120960:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a3e lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016150.121020:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334052736:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.121513:0:1983:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1734016150.121524:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334052864:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.125613:0:4742:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335465472 00000100:00100000:3.0:1734016150.125623:0:4742:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335465472 00000100:00100000:3.0:1734016150.125664:0:4742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13012 00000100:00100000:3.0:1734016150.125671:0:4742:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335465472:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.125687:0:4742:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.125698:0:4742:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.125727:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.125729:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016150 atime: 1734016149 ctime: 1734016150 blocks: 32 00000020:00010000:3.0:1734016150.125733:0:4742:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016150.125734:0:4742:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016150 atime: 1734016149 ctime: 1734016150 blocks: 32 00000100:00100000:3.0:1734016150.125782:0:4742:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a2300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7682:x1818247335465472:12345-192.168.201.116@tcp:106: Request processed in 111us (518us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.125789:0:4742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13012 00010000:00010000:3.0:1734016150.128170:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.128192:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.128206:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.128212:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.128218:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016150.128233:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3158904, setting rss=3158904; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016150.128555:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.128563:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.128570:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.128577:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa367cc0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016150.128582:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.128598:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016150.128603:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.128608:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa367cc0/0xa50fd972cb8e7a45 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016150.128643:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334052864:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.129209:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016150.129220:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334052992:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.132245:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335465600 00000100:00100000:3.0:1734016150.132254:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335465600 00000100:00100000:3.0:1734016150.132669:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13013 00000100:00100000:3.0:1734016150.132676:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335465600:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.132691:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.132703:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.132730:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.132732:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016150 atime: 1734016149 ctime: 1734016150 blocks: 32 00000020:00010000:3.0:1734016150.132735:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016150.132736:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016150 atime: 1734016149 ctime: 1734016150 blocks: 32 00000100:00100000:3.0:1734016150.132926:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335465600:12345-192.168.201.116@tcp:106: Request processed in 250us (1205us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.132933:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13013 00010000:00010000:3.0:1734016150.147811:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.147840:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.147859:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.147866:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.147873:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016150.147889:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3158904, setting rss=3158904; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016150.147907:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.147914:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.147921:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.147928:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366640 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016150.147934:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.147946:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016150.147953:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.147959:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366640/0xa50fd972cb8e7a4c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016150.148013:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334052992:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.148859:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334053120:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.159070:0:4741:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1734016150.159090:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335465728 00000100:00100000:3.0:1734016150.159099:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335465728 00000100:00100000:3.0:1734016150.159359:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13014 00000100:00100000:3.0:1734016150.159368:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012e2dc050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335465728:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.159383:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.159394:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.159422:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5cd48[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.159425:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016150 atime: 1734016149 ctime: 1734016150 blocks: 32 00000020:00010000:3.0:1734016150.159429:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800ab9f24b0[0x4, 3, [0x280000400:0x8:0x0] hash] 00000020:00010000:3.0:1734016150.159430:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 3158904 mtime: 1734016150 atime: 1734016149 ctime: 1734016150 blocks: 32 00000100:00100000:3.0:1734016150.159606:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012e2dc050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335465728:12345-192.168.201.116@tcp:106: Request processed in 237us (1751us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.159613:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13014 00010000:00010000:3.0:1734016150.162866:0:1984:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.162886:0:1984:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.162899:0:1984:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.162905:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.162910:0:1984:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016150.162924:0:1984:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3158904, setting rss=3158904; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016150.162969:0:1984:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.162976:0:1984:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.162992:0:1984:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.163000:0:1984:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa366ac0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016150.163004:0:1984:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.163014:0:1984:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016150.163019:0:1984:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.163025:0:1984:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff88012a5ff800 lock: ffff8800aa366ac0/0xa50fd972cb8e7a53 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016150.163658:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334053120:192.168.201.116@tcp:101:fsx.0 00000100:00100000:3.0:1734016150.169502:0:4741:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247335465856 00000100:00100000:3.0:1734016150.169513:0:4741:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247335465856 00000100:00100000:3.0:1734016150.169557:0:4741:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.116@tcp, seq: 13015 00000100:00100000:3.0:1734016150.169565:0:4741:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335465856:12345-192.168.201.116@tcp:106: 00010000:00010000:3.0:1734016150.169583:0:4741:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.169593:0:4741:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff88012a5ff800 lock: ffff8800aa348fc0/0xa50fd972cb8e79e3 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117e7a expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000020:00010000:3.0:1734016150.171500:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ce10[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:3.0:1734016150.171504:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016150 atime: 1734016149 ctime: 1734016150 blocks: 232 00000020:00010000:3.0:1734016150.171512:0:4741:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b3a5ced8[0x4, 2, [0x240000400:0x6:0x0] hash] 00000020:00010000:3.0:1734016150.171514:0:4741:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016150 atime: 1734016149 ctime: 1734016150 blocks: 232 00000100:00100000:3.0:1734016150.171602:0:4741:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a78a0380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7682:x1818247335465856:12345-192.168.201.116@tcp:106: Request processed in 2034us (2234us total) trans 0 rc 0/0 00000100:00100000:3.0:1734016150.171612:0:4741:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.116@tcp, seq: 13015 00010000:00010000:3.0:1734016150.177377:0:1983:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.177394:0:1983:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.177406:0:1983:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.177412:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.177417:0:1983:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 5/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016150.177429:0:1983:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:3.0:1734016150.177452:0:1983:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 4/1,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.177459:0:1983:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.177478:0:1983:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 4/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.177485:0:1983:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800aa3646c0 already unused, calling callback (ffffffffa0a12d90) 00010000:00010000:3.0:1734016150.177490:0:1983:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 5/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.177499:0:1983:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1734016150.177503:0:1983:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.177509:0:1983:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800aa3646c0/0xa50fd972cb8e7a5a lrc: 0/0,0 mode: --/PR res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016150.177546:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334053248:192.168.201.116@tcp:101:fsx.0 00000008:00010000:3.0:1734016150.178801:0:1984:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800b6ee76c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.178829:0:1984:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1671168 1675263) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016150.178867:0:1984:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982a00 - object 0x280000400:8 offset 1642496<>1708032 00000100:00100000:3.0:1734016150.178883:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016150.178894:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334053504:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016150.196234:0:1984:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800b6ee76c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00010000:3.0:1734016150.196555:0:1984:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800b6ee76c0 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000100:00100000:3.0:1734016150.196584:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334053504:192.168.201.116@tcp:3:fsx.0 00000008:00010000:3.0:1734016150.200104:0:10406:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800b6ee7f00 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016150.200143:0:10406:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1671168 1675263) ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800020000020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00000008:00100000:3.0:1734016150.200215:0:10406:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a7982a00 - object 0x280000400:8 offset 942080<>1679360 00000100:00100000:0.0:1734016151.761249:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1982:1818247334053760:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016151.761305:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334054016:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016151.761321:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334054272:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016151.761334:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334054528:192.168.201.116@tcp:400:kworker.0 00000100:00100000:3.0:1734016155.072792:0:1983:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334054656:192.168.201.116@tcp:4:fsx.0 00000100:00100000:0.0:1734016156.768747:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78f8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334054912:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016156.768817:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7913480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334055168:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016156.768838:0:1982:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7913b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334055424:192.168.201.116@tcp:400:kworker.0 00000080:00200000:3.0:1734016161.248608:0:9727:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016161.248628:0:9727:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016161.248670:0:9727:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1734016161.248681:0:9727:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1734016161.248685:0:9727:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1734016161.248692:0:9727:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1734016161.776771:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1984:1818247334055680:192.168.201.116@tcp:400:kworker.0 00000100:00100000:3.0:1734016161.776851:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334055936:192.168.201.116@tcp:400:kworker.0 00000100:00100000:3.0:1734016161.776873:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334056192:192.168.201.116@tcp:400:kworker.0 00000100:00100000:3.0:1734016161.776891:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334056448:192.168.201.116@tcp:400:kworker.0 00000100:00080000:0.0:1734016166.802545:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016166.802553:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016166.802557:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016166.802567:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 143 00000100:00080000:0.0:1734016166.802578:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016166.802590:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016166.802613:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334056960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016166.802702:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334057088:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016166.802762:0:1981:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334057344:192.168.201.116@tcp:400:kworker.0 00000100:00000400:0.0:1734016167.802569:0:1981:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016151/real 1734016151] req@ffff8800a78fa300 x1818247334053888/t0(0) o400->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016167.802847:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1984:1818247334055680:192.168.201.116@tcp:400:kworker.0 00000100:00080000:3.0:1734016167.802907:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016167.802916:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016167.802920:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016167.802928:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 101 00000100:00080000:3.0:1734016167.802943:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734016167.802977:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334057472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016167.803040:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016167.803044:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016167.803046:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016167.803077:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 146 00000100:00080000:3.0:1734016167.803084:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:3.0:1734016167.803094:0:1984:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1734016167.803110:0:1984:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334057600:192.168.201.116@tcp:400:kworker.0 00000008:00100000:3.0:1734016167.803207:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734016167.803230:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334057728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016167.803268:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016167.803274:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016167.803278:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016167.803283:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016167.803286:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016167.803292:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 127 00000100:00080000:3.0:1734016167.803303:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734016167.803312:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016167.803330:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334057856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016167.803382:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016167.803387:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016167.803390:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016167.803399:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 127 00000100:00080000:3.0:1734016167.803410:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734016167.803431:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016167.803447:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334058112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016167.803534:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016167.803538:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016167.803541:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016167.803544:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1734016167.842314:0:1981:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a78fa300 x1818247334053888/t0(0) o400->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016167 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016167.842327:0:1981:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1734016167.842331:0:1981:0:(import.c:200:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff88012a571000: Connection to lustre-MDT0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1734016167.842335:0:1981:0:(import.c:432:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:0.0:1734016167.842355:0:1981:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016151/real 1734016151] req@ffff8800a78faa00 x1818247334054144/t0(0) o400->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016167.842369:0:1981:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a78faa00 x1818247334054144/t0(0) o400->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016167 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016167.842376:0:1981:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff8800aa959000 already not connected (conn 3, was 2): CONNECTING 00000100:00000400:0.0:1734016167.842382:0:1981:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016151/real 1734016151] req@ffff8800a78f9880 x1818247334054400/t0(0) o400->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016167.842393:0:1981:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a78f9880 x1818247334054400/t0(0) o400->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016167 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016167.842401:0:1981:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1734016167.842404:0:1981:0:(import.c:200:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff88012a5ff800: Connection to lustre-OST0000 (at 192.168.201.116@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1734016167.842407:0:1981:0:(import.c:432:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00100000:0.0:1734016167.842422:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1981:1818247334054784:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016167.842429:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334053888:192.168.201.116@tcp:400:kworker.0 00000100:00000400:0.0:1734016172.802590:0:1982:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016156/real 1734016156] req@ffff8800a78f8380 x1818247334054912/t0(0) o400->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016172.802663:0:1982:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a78f8380 x1818247334054912/t0(0) o400->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016172.802674:0:1982:0:(import.c:232:ptlrpc_set_import_discon()) mdc: import ffff8800b6116800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1734016172.802682:0:1982:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016156/real 1734016156] req@ffff8800a7913480 x1818247334055168/t0(0) o400->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016172.802694:0:1982:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a7913480 x1818247334055168/t0(0) o400->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016172.802701:0:1982:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff8800aa959000 already not connected (conn 3, was 2): CONNECTING 00000100:00000400:0.0:1734016172.802706:0:1982:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016156/real 1734016156] req@ffff8800a7913b80 x1818247334055424/t0(0) o400->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016172.802716:0:1982:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a7913b80 x1818247334055424/t0(0) o400->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016172.802723:0:1982:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff88012e28c800 already not connected (conn 3, was 2): CONNECTING 00000100:00100000:0.0:1734016172.802735:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334054912:192.168.201.116@tcp:400:kworker.0 00000100:00000400:0.0:1734016172.841580:0:1981:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016156/real 1734016156] req@ffff8800a7912d80 x1818247334055040/t0(0) o400->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016172.841633:0:1981:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a7912d80 x1818247334055040/t0(0) o400->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016172.841642:0:1981:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff8800b6117800 already not connected (conn 3, was 2): CONNECTING 00000100:00000400:0.0:1734016172.841649:0:1981:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016156/real 1734016156] req@ffff8800a7911500 x1818247334055296/t0(0) o400->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016172.841661:0:1981:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a7911500 x1818247334055296/t0(0) o400->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016172.841668:0:1981:0:(import.c:232:ptlrpc_set_import_discon()) mdc: import ffff88012e28f000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1734016172.841672:0:1981:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016156/real 1734016156] req@ffff8800a7913800 x1818247334055552/t0(0) o400->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016172.841684:0:1981:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a7913800 x1818247334055552/t0(0) o400->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016172 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016172.841691:0:1981:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff88012e28b000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:0.0:1734016172.841704:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7911500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334055296:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016172.841710:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7913800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334055552:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016176.842782:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334054144:192.168.201.116@tcp:400:kworker.0 00000100:00080000:3.0:1734016176.842793:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016176.842801:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016176.842804:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016176.842808:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016176.842811:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734016176.842814:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7913480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334055168:192.168.201.116@tcp:400:kworker.0 00000100:00080000:3.0:1734016176.842814:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016176.842818:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016176.842821:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016176.842824:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016176.842833:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 438 00000100:00080000:3.0:1734016176.842843:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1734016176.842850:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734016176.842862:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016176.842891:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334058496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016176.842926:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016176.842930:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016176.842933:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016176.842937:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016176.842939:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016176.842942:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:3.0:1734016177.802611:0:1984:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016161/real 1734016161] req@ffff8800a78a1880 x1818247334055936/t0(0) o400->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016177.821025:0:1984:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a78a1880 x1818247334055936/t0(0) o400->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016177.821037:0:1984:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff8800b6117800 already not connected (conn 3, was 2): CONNECTING 00000100:00000400:3.0:1734016177.821044:0:1984:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016161/real 1734016161] req@ffff8800a78a2d80 x1818247334056192/t0(0) o400->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016177.821065:0:1984:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a78a2d80 x1818247334056192/t0(0) o400->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016177.821072:0:1984:0:(import.c:232:ptlrpc_set_import_discon()) mdc: import ffff88012e28f000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1734016177.821077:0:1984:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016161/real 1734016161] req@ffff8800a78a0a80 x1818247334056448/t0(0) o400->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016177.821088:0:1984:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a78a0a80 x1818247334056448/t0(0) o400->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016177.821094:0:1984:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff88012e28b000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:3.0:1734016177.821105:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334056192:192.168.201.116@tcp:400:kworker.0 00000100:00100000:3.0:1734016177.821111:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1984:1818247334056448:192.168.201.116@tcp:400:kworker.0 00000100:00000400:3.0:1734016177.824726:0:1983:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016161/real 1734016161] req@ffff8800a78a0700 x1818247334055808/t0(0) o400->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016177.824752:0:1983:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a78a0700 x1818247334055808/t0(0) o400->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016177.824761:0:1983:0:(import.c:232:ptlrpc_set_import_discon()) mdc: import ffff8800b6116800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1734016177.824767:0:1983:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016161/real 1734016161] req@ffff8800a78a2680 x1818247334056064/t0(0) o400->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016177.824778:0:1983:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a78a2680 x1818247334056064/t0(0) o400->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016177.824785:0:1983:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff8800aa959000 already not connected (conn 4, was 2): CONNECTING 00000100:00000400:3.0:1734016177.824790:0:1983:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016161/real 1734016161] req@ffff8800a78a1500 x1818247334056320/t0(0) o400->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016177.824801:0:1983:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a78a1500 x1818247334056320/t0(0) o400->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016177 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016177.824808:0:1983:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff88012e28c800 already not connected (conn 3, was 2): CONNECTING 00000100:00100000:3.0:1734016177.824819:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334055808:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016177.842845:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7913b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334055424:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016177.842860:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78fad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334054016:192.168.201.116@tcp:400:kworker.0 00000100:00080000:3.0:1734016177.842862:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016177.842870:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016177.842873:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016177.842883:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 439 00000100:00080000:3.0:1734016177.842893:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1734016177.842900:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016177.842902:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7912d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8c598caf-7141-4c19-8566-0b7731516db9:1981:1818247334055040:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016177.842909:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334054400:192.168.201.116@tcp:400:kworker.0 00000100:00100000:3.0:1734016177.842928:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334058624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016177.842950:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016177.842954:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016177.842956:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016177.842960:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 439 00000100:00080000:3.0:1734016177.842963:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1734016177.842967:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734016177.842975:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016177.842986:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334058752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016177.842996:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016177.843000:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016177.843002:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016177.843005:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 439 00000100:00080000:3.0:1734016177.843019:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1734016177.843023:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734016177.843027:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734016177.843037:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334058880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016177.843047:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016177.843051:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016177.843053:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016177.843056:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016177.843057:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016177.843061:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 439 00000100:00080000:3.0:1734016177.843064:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1734016177.843068:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734016177.843072:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016177.843079:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334059008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016177.843103:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016177.843106:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016177.843108:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016177.843111:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 439 00000100:00080000:3.0:1734016177.843115:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1734016177.843119:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734016177.843123:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016177.843130:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334059136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016177.843139:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016177.843142:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016177.843143:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016177.843146:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 439 00000100:00080000:3.0:1734016177.843149:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 5s 00000100:00100000:0.0:1734016177.843150:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334059008:192.168.201.116@tcp:38:kworker.0 00000100:00080000:3.0:1734016177.843152:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734016177.843156:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016177.843163:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334059264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016177.843209:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334059136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016177.843239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334059264:192.168.201.116@tcp:8:kworker.0 00000100:00080000:3.0:1734016181.856575:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016181.856583:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016181.856587:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016181.856590:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016181.856593:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016181.856596:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016181.856599:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016181.856602:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016181.856605:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016181.856608:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016181.856611:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016181.856614:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016181.856616:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016181.856619:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:0.0:1734016182.802618:0:1982:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016166/real 1734016166] req@ffff8800a934ea00 x1818247334057216/t0(0) o400->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016182.821262:0:1982:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a934ea00 x1818247334057216/t0(0) o400->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016182 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016182.821275:0:1982:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff88012e28c800 already not connected (conn 4, was 2): CONNECTING 00000100:00000400:0.0:1734016182.821284:0:1982:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016166/real 1734016166] req@ffff8800a934d500 x1818247334056704/t0(0) o400->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016182.821298:0:1982:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a934d500 x1818247334056704/t0(0) o400->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016182 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016182.821305:0:1982:0:(import.c:232:ptlrpc_set_import_discon()) mdc: import ffff8800b6116800 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:0.0:1734016182.821310:0:1982:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016166/real 1734016166] req@ffff8800a934e300 x1818247334056832/t0(0) o400->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016182.821322:0:1982:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a934e300 x1818247334056832/t0(0) o400->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016182 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016182.821329:0:1982:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff8800b6117800 already not connected (conn 4, was 2): CONNECTING 00000100:00100000:0.0:1734016182.821342:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334056704:192.168.201.116@tcp:400:kworker.0 00000100:00000400:0.0:1734016182.841629:0:1981:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016166/real 1734016166] req@ffff8800a934c380 x1818247334057088/t0(0) o400->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016182.841655:0:1981:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a934c380 x1818247334057088/t0(0) o400->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016182 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016182.841664:0:1981:0:(import.c:232:ptlrpc_set_import_discon()) mdc: import ffff88012e28f000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:0.0:1734016182.841671:0:1981:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016166/real 1734016166] req@ffff8800a934fb80 x1818247334057344/t0(0) o400->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016182.841683:0:1981:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a934fb80 x1818247334057344/t0(0) o400->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016182 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016182.841690:0:1981:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff88012e28b000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1734016182.841700:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334057088:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016182.841707:0:1981:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1981:1818247334057344:192.168.201.116@tcp:400:kworker.0 00000100:00080000:3.0:1734016182.858568:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016182.858576:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016182.858579:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016182.858582:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016182.858585:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016182.858588:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016182.858591:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016182.858594:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016182.858596:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016182.858599:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016182.858601:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016182.858604:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016182.858606:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016182.858610:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:3.0:1734016183.820614:0:1984:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016167/real 1734016167] req@ffff8800a7980e00 x1818247334057600/t0(0) o400->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016183 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016183.820641:0:1984:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a7980e00 x1818247334057600/t0(0) o400->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016183 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016183.820651:0:1984:0:(import.c:232:ptlrpc_set_import_discon()) mdc: import ffff8800b6116800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:3.0:1734016183.820663:0:1984:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8c598caf-7141-4c19-8566-0b7731516db9:1984:1818247334057600:192.168.201.116@tcp:400:kworker.0 00000100:00000400:3.0:1734016183.824611:0:1983:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016167/real 1734016167] req@ffff8800a7983800 x1818247334057984/t0(0) o400->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016183 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016183.824636:0:1983:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a7983800 x1818247334057984/t0(0) o400->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016183 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016183.824646:0:1983:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff88012e28c800 already not connected (conn 4, was 2): CONNECTING 00000100:00080000:3.0:1734016187.872589:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016187.872597:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016187.872601:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016187.872604:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016187.872607:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016187.872610:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016187.872613:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016187.872616:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016187.872619:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016187.872623:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016187.872625:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016187.872628:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1734016187.872631:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016187.872634:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1734016191.842579:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016176/real 1734016176] req@ffff8800a7982300 x1818247334058496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016191.842612:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334058496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016191 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016191.842647:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016191.842665:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016191.842671:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334058496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016191.842750:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8c598caf-7141-4c19-8566-0b7731516db9:1983:1818247334056064:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016191.842943:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7910000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334059392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016192.842567:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016177/real 1734016177] req@ffff8800a7981f80 x1818247334059264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842601:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334059264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842613:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016177/real 1734016177] req@ffff8800a7980a80 x1818247334059008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842626:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334059008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842635:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016177/real 1734016177] req@ffff8800a7981880 x1818247334059136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842645:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334059136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842654:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016177/real 1734016177] req@ffff8800a7982680 x1818247334058880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842671:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334058880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842680:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016177/real 1734016177] req@ffff8800a7981500 x1818247334058752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842691:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334058752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842699:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016177/real 1734016177] req@ffff8800a7982d80 x1818247334058624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016192.842711:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334058624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016192 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016192.842731:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016192.842755:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016192.842761:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334059264:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016192.842767:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016192.842772:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016192.842776:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334059008:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734016192.842794:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016192.842798:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016192.842802:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334059136:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016192.842808:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016192.842811:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016192.842815:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334058880:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016192.842819:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016192.842823:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016192.842826:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334058752:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734016192.842830:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016192.842834:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016192.842837:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334058624:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016192.842878:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1982:1818247334057216:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016192.842884:0:1982:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8c598caf-7141-4c19-8566-0b7731516db9:1982:1818247334056832:192.168.201.116@tcp:400:kworker.0 00000100:00100000:3.0:1734016192.842900:0:1983:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1983:1818247334057984:192.168.201.116@tcp:400:kworker.0 00000100:00100000:0.0:1734016192.842959:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7910700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334059520:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016192.843028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7913100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334059648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016192.843050:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7911f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334059776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016192.843092:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7912680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334059904:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016192.843134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7910a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334060032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016192.843177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7912300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334060160:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016196.848630:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016196.848637:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016196.848641:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016196.848645:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016196.848647:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016196.848650:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016196.848653:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016196.848656:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016196.848659:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016196.848662:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016196.848664:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016196.848667:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016196.848671:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016196.848674:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016197.850614:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016197.850627:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016197.850630:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016197.850634:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016197.850637:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016197.850640:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016197.850642:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016197.850645:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016197.850648:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016197.850651:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016197.850653:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016197.850656:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016197.850658:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016197.850661:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016202.864612:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016202.864624:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016202.864627:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016202.864632:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016202.864635:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016202.864639:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016202.864641:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016202.864645:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016202.864647:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016202.864650:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016202.864653:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016202.864656:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016202.864659:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016202.864662:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000800:00000100:2.1:1734016205.557638:0:1972:0:(socklnd_cb.c:2434:ksocknal_find_timed_out_conn()) Timeout sending data to 12345-192.168.201.116@tcp (192.168.201.116:988) the network or that node may be down. 00000100:00080000:0.0:1734016207.872624:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016207.872635:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016207.872639:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016207.872643:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016207.872646:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016207.872650:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016207.872652:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016207.872656:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016207.872661:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016207.872664:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016207.872667:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016207.872670:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016207.872673:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016207.872676:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016211.842789:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016211.842797:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016211.842801:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016211.842804:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016211.842807:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016211.842810:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016211.842813:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016211.842816:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016211.842820:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016211.842827:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 463 00000100:00080000:0.0:1734016211.842837:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1734016211.842845:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016211.842856:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016211.842886:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334060288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016211.842928:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016211.842933:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016211.842935:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016211.842939:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016211.842941:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016211.842945:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016212.842903:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016212.842911:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016212.842915:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016212.842923:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 464 00000100:00080000:0.0:1734016212.842956:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1734016212.842964:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016212.842993:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334060416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016212.843026:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016212.843030:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016212.843032:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016212.843036:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 464 00000100:00080000:0.0:1734016212.843040:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1734016212.843044:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016212.843053:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016212.843067:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334060544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016212.843078:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016212.843081:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016212.843083:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016212.843086:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 464 00000100:00080000:0.0:1734016212.843089:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1734016212.843092:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016212.843096:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016212.843105:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334060672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016212.843113:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016212.843117:0:9727:0:(pinger.c:272:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1734016212.843119:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016212.843122:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016212.843124:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016212.843127:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 464 00000100:00080000:0.0:1734016212.843131:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1734016212.843134:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016212.843138:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016212.843145:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334060800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016212.843154:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016212.843157:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016212.843159:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016212.843162:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 464 00000100:00080000:0.0:1734016212.843165:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1734016212.843168:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016212.843172:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016212.843181:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334060928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016212.843192:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016212.843195:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016212.843196:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016212.843199:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 464 00000100:00080000:0.0:1734016212.843202:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1734016212.843205:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016212.843209:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016212.843217:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334061056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016214.352988:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016212/real 1734016214] req@ffff8800a934c000 x1818247334061056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016214.353017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334061056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016214.353029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016214.353033:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016214.353038:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334061056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016214.353046:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016212/real 1734016214] req@ffff8800a934df80 x1818247334060928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016214.353061:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334060928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016214.353074:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016214.353078:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016214.353084:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334060928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016214.353093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016212/real 1734016214] req@ffff8800a934d500 x1818247334060800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016214.353110:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334060800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016214.353122:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016214.353125:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016214.353131:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334060800:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016214.353140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016212/real 1734016214] req@ffff8800a934fb80 x1818247334060544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016214.353158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334060544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016214.353170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016214.353188:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016214.353194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334060544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016214.353205:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016212/real 1734016214] req@ffff8800a934c380 x1818247334060672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016214.353226:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334060672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016214.353238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016214.353242:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016214.353247:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334060672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016214.353257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016212/real 1734016214] req@ffff8800a934ea00 x1818247334060416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016214.353275:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334060416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016214.353286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016214.353289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016214.353293:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334060416:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016214.353300:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016211/real 1734016214] req@ffff8800a934e300 x1818247334060288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016214.353312:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334060288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016214.353319:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016214.353322:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016214.353325:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334060288:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016216.848570:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016216.848579:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016216.848583:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016216.848594:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 484 00000100:00080000:0.0:1734016216.848603:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016216.848611:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016216.848645:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334061184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016216.848679:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016216.848683:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016216.848685:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016216.848690:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 484 00000100:00080000:0.0:1734016216.848693:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016216.848697:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016216.848707:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016216.848718:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334061312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016216.848728:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016216.848731:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016216.848732:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016216.848736:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 484 00000100:00080000:0.0:1734016216.848739:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016216.848742:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016216.848747:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016216.848755:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334061440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016216.848764:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016216.848767:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016216.848768:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016216.848771:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 483 00000100:00080000:0.0:1734016216.848774:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016216.848778:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016216.848782:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016216.848790:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334061184:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016216.848792:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334061568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016216.848801:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016216.848804:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016216.848805:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016216.848808:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 484 00000100:00080000:0.0:1734016216.848811:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016216.848814:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016216.848817:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016216.848825:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334061696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016216.848833:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016216.848836:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016216.848837:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016216.848841:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 484 00000100:00080000:0.0:1734016216.848844:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016216.848847:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016216.848851:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016216.848858:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334061824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016216.848900:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1734016216.848903:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334061312:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734016216.848904:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016216.848906:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016216.848910:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 484 00000100:00080000:0.0:1734016216.848913:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016216.848916:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016216.848921:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016216.848925:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334061440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016216.848930:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334061952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016216.848948:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334061568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016216.848963:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334061696:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016216.848978:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334061824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016216.848993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334061952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016217.359046:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016216/real 1734016217] req@ffff8800a934f480 x1818247334061184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016217.359082:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334061184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016217.359102:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016217.359108:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016217.359116:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334061184:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016217.359128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016216/real 1734016217] req@ffff8800a934e680 x1818247334061312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016217.359148:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334061312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016217.359161:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016217.359165:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016217.359171:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334061312:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016217.359182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016216/real 1734016217] req@ffff8800a934f100 x1818247334061440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016217.359201:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334061440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016217.359213:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016217.359218:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016217.359223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334061440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016217.359259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016216/real 1734016217] req@ffff8800a934ce00 x1818247334061568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016217.359280:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334061568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016217.359293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016217.359297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016217.359303:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334061568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016217.359314:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016216/real 1734016217] req@ffff8800a934e300 x1818247334061696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016217.359332:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334061696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016217.359344:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016217.359348:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016217.359353:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334061696:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016217.359363:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016216/real 1734016217] req@ffff8800a934ea00 x1818247334061824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016217.359394:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334061824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016217.359407:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016217.359411:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016217.359416:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334061824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016217.359426:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016216/real 1734016217] req@ffff8800a934c380 x1818247334061952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016217.359444:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334061952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016217.359455:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016217.359459:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016217.359532:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334061952:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016221.856594:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016221.856603:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016221.856607:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016221.856618:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 488 00000100:00080000:0.0:1734016221.856628:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016221.856635:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016221.856672:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334062080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016221.856708:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016221.856715:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016221.856717:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016221.856722:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 488 00000100:00080000:0.0:1734016221.856725:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016221.856729:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016221.856739:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016221.856750:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334062208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016221.856760:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016221.856763:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016221.856765:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016221.856769:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 488 00000100:00080000:0.0:1734016221.856772:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016221.856776:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016221.856805:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016221.856816:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334062336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016221.856826:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016221.856829:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016221.856830:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016221.856834:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 488 00000100:00080000:0.0:1734016221.856837:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016221.856840:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016221.856845:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016221.856855:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334062464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016221.856864:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016221.856867:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016221.856869:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016221.856873:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 488 00000100:00080000:0.0:1734016221.856875:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016221.856879:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016221.856883:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016221.856891:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334062592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016221.856899:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016221.856901:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016221.856903:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016221.856906:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 488 00000100:00080000:0.0:1734016221.856909:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016221.856912:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016221.856916:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016221.856923:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334062720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016221.856931:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016221.856934:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016221.856936:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016221.856939:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 488 00000100:00080000:0.0:1734016221.856943:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016221.856945:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016221.856949:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016221.856957:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334062848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016223.371027:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016221/real 1734016223] req@ffff8800a934ce00 x1818247334062464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016223.371060:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334062464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016223.371073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016223.371077:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016223.371083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334062464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016223.371091:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016221/real 1734016223] req@ffff8800a934f100 x1818247334062592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016223.371104:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334062592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016223.371111:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016223.371114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016223.371118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334062592:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016223.371124:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016221/real 1734016223] req@ffff8800a934e680 x1818247334062720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016223.371135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334062720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016223.371144:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016223.371149:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016223.371155:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334062720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016223.371166:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016221/real 1734016223] req@ffff8800a934f480 x1818247334062848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016223.371184:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334062848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016223.371196:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016223.371213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016223.371220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334062848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016223.371231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016221/real 1734016223] req@ffff8800a934c380 x1818247334062080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016223.371251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334062080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016223.371264:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016223.371268:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016223.371274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334062080:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016223.371284:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016221/real 1734016223] req@ffff8800a934ea00 x1818247334062208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016223.371303:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334062208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016223.371316:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016223.371321:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016223.371326:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334062208:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016223.371336:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016221/real 1734016223] req@ffff8800a934e300 x1818247334062336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016223.371355:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334062336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016223.371366:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016223.371370:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016223.371376:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334062336:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016226.864577:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016226.864586:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016226.864591:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016226.864601:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 493 00000100:00080000:0.0:1734016226.864610:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016226.864617:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016226.864654:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334062976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016226.864689:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016226.864693:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016226.864733:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016226.864738:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 493 00000100:00080000:0.0:1734016226.864741:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016226.864746:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016226.864756:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016226.864769:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334063104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016226.864786:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016226.864790:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016226.864791:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016226.864795:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 493 00000100:00080000:0.0:1734016226.864798:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016226.864801:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016226.864806:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016226.864816:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334063232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016226.864822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334062976:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016226.864826:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016226.864829:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016226.864831:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016226.864834:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 493 00000100:00080000:0.0:1734016226.864837:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016226.864840:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016226.864844:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016226.864853:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334063360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016226.864862:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016226.864865:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016226.864866:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016226.864869:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 493 00000100:00080000:0.0:1734016226.864873:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016226.864876:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016226.864880:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016226.864888:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334063488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016226.864895:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016226.864898:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016226.864900:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016226.864903:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 493 00000100:00080000:0.0:1734016226.864906:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016226.864909:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016226.864913:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016226.864918:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016226.864921:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334063616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016226.864929:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016226.864932:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016226.864933:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016226.864936:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 493 00000100:00080000:0.0:1734016226.864939:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016226.864942:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016226.864946:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016226.864946:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016226/real 1734016226] req@ffff8800a934fb80 x1818247334062976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016226.864953:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334063744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016226.864968:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334062976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016226.864981:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016226.864985:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016226.864990:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334062976:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016226.864999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334063104:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016226.865017:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016226.865024:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016226/real 1734016226] req@ffff8800a934d500 x1818247334063104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016226.865035:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334063104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016226.865044:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016226.865047:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016226.865050:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334063104:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016226.865079:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334063232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016226.865094:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016226.865100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016226/real 1734016226] req@ffff8800a934df80 x1818247334063232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016226.865111:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334063232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016226.865120:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016226.865122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016226.865125:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334063232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016226.865132:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334063360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016226.865145:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016226.865151:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016226/real 1734016226] req@ffff8800a934c000 x1818247334063360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016226.865161:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334063360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016226.865168:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016226.865171:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016226.865174:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334063360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016226.865181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334063488:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016226.865195:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016226.865202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016226/real 1734016226] req@ffff8800a934d180 x1818247334063488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016226.865212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334063488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016226.865219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016226.865221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016226.865224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334063488:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016226.865230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334063616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016226.865243:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016226.865249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016226/real 1734016226] req@ffff8800a934dc00 x1818247334063616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016226.865259:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334063616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016226.865267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016226.865269:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016226.865273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334063616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016226.865278:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334063744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016226.865292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016226.865298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016226/real 1734016226] req@ffff8800a934d880 x1818247334063744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016226.865308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334063744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016226.865315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016226.865318:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016226.865321:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334063744:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016231.872647:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016231.872661:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016231.872666:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016231.872678:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 498 00000100:00080000:0.0:1734016231.872687:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016231.872694:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016231.872731:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334063872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016231.872767:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016231.872772:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016231.872775:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016231.872782:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 498 00000100:00080000:0.0:1734016231.872787:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016231.872792:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016231.872805:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016231.872823:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334064000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016231.872835:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334063872:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016231.872840:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016231.872844:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016231.872846:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016231.872852:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 498 00000100:00080000:0.0:1734016231.872856:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016231.872860:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016231.872868:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016231.872881:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334064128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016231.872893:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016231.872898:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016231.872900:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016231.872905:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 498 00000100:00080000:0.0:1734016231.872910:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016231.872914:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016231.872920:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016231.872936:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334064256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016231.872942:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334064000:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734016231.872950:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016231.872955:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016231.872958:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016231.872962:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 498 00000100:00080000:0.0:1734016231.872967:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1734016231.872971:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334064128:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016231.872972:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016231.872979:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016231.872991:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334064384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016231.872997:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334064256:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016231.873006:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016231.873010:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016231.873014:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016231.873020:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 498 00000100:00100000:3.0:1734016231.873023:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334064384:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734016231.873024:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016231.873029:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016231.873035:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016231.873048:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334064512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016231.873065:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016231.873070:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016231.873073:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016231.873078:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 498 00000100:00100000:3.0:1734016231.873082:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334064512:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016231.873083:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016231.873088:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016231.873094:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016231.873107:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334064640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016232.389013:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016231/real 1734016232] req@ffff8800a934c000 x1818247334064640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016232.389051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334064640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016232.389065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016232.389070:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016232.389076:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334064640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016232.389085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016231/real 1734016232] req@ffff8800a934d180 x1818247334064512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016232.389099:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334064512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016232.389107:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016232.389110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016232.389113:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334064512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016232.389120:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016231/real 1734016232] req@ffff8800a934dc00 x1818247334064384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016232.389131:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334064384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016232.389138:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016232.389142:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016232.389147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334064384:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016232.389157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016231/real 1734016232] req@ffff8800a934d880 x1818247334064256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016232.389175:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334064256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016232.389187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016232.389190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016232.389195:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334064256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016232.389206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016231/real 1734016232] req@ffff8800a934c700 x1818247334064000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016232.389236:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334064000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016232.389248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016232.389253:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016232.389258:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334064000:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016232.389269:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016231/real 1734016232] req@ffff8800a934ed80 x1818247334064128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016232.389288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334064128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016232.389301:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016232.389306:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016232.389311:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334064128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016232.389322:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016231/real 1734016232] req@ffff8800a934ca80 x1818247334063872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016232.389340:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334063872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016232.389351:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016232.389355:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016232.389361:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334063872:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016236.880608:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016236.880622:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016236.880627:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016236.880637:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 503 00000100:00080000:0.0:1734016236.880646:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016236.880653:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016236.880691:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334064768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016236.880731:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016236.880736:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016236.880738:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016236.880742:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 503 00000100:00080000:0.0:1734016236.880745:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016236.880749:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016236.880759:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016236.880770:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334064896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016236.880779:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016236.880782:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016236.880784:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016236.880788:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 503 00000100:00080000:0.0:1734016236.880791:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016236.880795:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016236.880799:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016236.880808:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334065024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016236.880816:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016236.880819:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016236.880820:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016236.880824:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 503 00000100:00080000:0.0:1734016236.880826:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016236.880830:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016236.880834:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016236.880842:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334065152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016236.880843:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334064768:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016236.880850:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016236.880853:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016236.880855:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016236.880858:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 503 00000100:00080000:0.0:1734016236.880861:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016236.880864:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016236.880868:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016236.880877:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334065280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016236.880885:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016236.880887:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016236.880889:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016236.880892:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 503 00000100:00080000:0.0:1734016236.880895:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016236.880898:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016236.880902:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016236.880909:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016236.880910:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334065408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016236.880918:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016236.880921:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016236.880923:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016236.880926:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 503 00000100:00080000:0.0:1734016236.880929:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1734016236.880931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016236/real 1734016236] req@ffff8800a934df80 x1818247334064768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016236.880932:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016236.880936:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016236.880943:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334065536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016236.880951:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334064768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016236.880962:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016236.880965:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016236.880970:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334064768:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016236.880977:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334064896:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016236.880994:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016236.881000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016236/real 1734016236] req@ffff8800a934d500 x1818247334064896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016236.881012:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334064896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016236.881019:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016236.881022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016236.881025:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334064896:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016236.881031:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334065024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016236.881044:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016236.881050:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016236/real 1734016236] req@ffff8800a934fb80 x1818247334065024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016236.881060:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334065024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016236.881074:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016236.881077:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016236.881080:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334065024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016236.881106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334065152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016236.881119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016236.881125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016236/real 1734016236] req@ffff8800a934e300 x1818247334065152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016236.881135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334065152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016236.881143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016236.881145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016236.881148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334065152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016236.881153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334065280:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016236.881165:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016236.881171:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016236/real 1734016236] req@ffff8800a934ea00 x1818247334065280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016236.881180:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334065280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016236.881187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016236.881190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016236.881193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334065280:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016236.881199:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334065408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016236.881211:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016236.881217:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016236/real 1734016236] req@ffff8800a934c380 x1818247334065408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016236.881227:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334065408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016236.881233:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016236.881236:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016236.881238:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334065408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016236.881244:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334065536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016236.881256:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016236.881261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016236/real 1734016236] req@ffff8800a934f480 x1818247334065536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016236.881271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334065536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016236.881278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016236.881280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016236.881283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334065536:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016241.888619:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016241.888631:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016241.888635:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016241.888646:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 508 00000100:00080000:0.0:1734016241.888655:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016241.888663:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016241.888700:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334065664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016241.888735:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016241.888739:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016241.888740:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016241.888745:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 508 00000100:00080000:0.0:1734016241.888748:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016241.888752:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016241.888762:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016241.888773:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334065792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016241.888783:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016241.888787:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016241.888788:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016241.888792:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 508 00000100:00080000:0.0:1734016241.888795:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016241.888798:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016241.888803:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016241.888811:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334065920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016241.888820:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016241.888823:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016241.888824:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016241.888828:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 508 00000100:00080000:0.0:1734016241.888830:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016241.888833:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016241.888837:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016241.888847:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334066048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016241.888847:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334065664:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016241.888856:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016241.888859:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016241.888860:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016241.888864:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 508 00000100:00080000:0.0:1734016241.888867:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016241.888870:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016241.888874:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016241.888882:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334066176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016241.888890:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016241.888893:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016241.888895:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016241.888898:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 508 00000100:00080000:0.0:1734016241.888900:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016241.888904:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016241.888908:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016241.888915:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334066304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016241.888922:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016241.888925:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016241.888927:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016241.888930:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 508 00000100:00080000:0.0:1734016241.888933:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016241.888936:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:3.0:1734016241.888939:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334065792:192.168.201.116@tcp:38:kworker.0 00000008:00100000:0.0:1734016241.888940:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016241.888947:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334066432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016241.888961:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334065920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016241.888986:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334066048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016241.889001:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334066176:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016241.889015:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334066304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016241.889029:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334066432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016244.412987:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016241/real 1734016244] req@ffff8800a934f480 x1818247334066048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016244.413016:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334066048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016244.413028:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016244.413032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016244.413037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334066048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016244.413045:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016241/real 1734016244] req@ffff8800a934c380 x1818247334066176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016244.413057:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334066176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016244.413067:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016244.413070:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016244.413076:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334066176:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016244.413086:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016241/real 1734016244] req@ffff8800a934ea00 x1818247334066304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016244.413114:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334066304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016244.413126:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016244.413130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016244.413135:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334066304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016244.413157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016241/real 1734016244] req@ffff8800a934e300 x1818247334066432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016244.413179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334066432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016244.413191:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016244.413195:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016244.413201:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334066432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016244.413211:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016241/real 1734016244] req@ffff8800a934e680 x1818247334065664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016244.413230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334065664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016244.413242:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016244.413246:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016244.413251:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334065664:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016244.413261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016241/real 1734016244] req@ffff8800a934f100 x1818247334065792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016244.413289:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334065792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016244.413301:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016244.413306:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016244.413311:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334065792:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016244.413321:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016241/real 1734016244] req@ffff8800a934ce00 x1818247334065920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016244.413338:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334065920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016244.413350:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016244.413354:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016244.413359:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334065920:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016246.896592:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016246.896604:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016246.896610:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016246.896623:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 513 00000100:00080000:0.0:1734016246.896636:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016246.896645:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016246.896694:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334066560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016246.896733:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016246.896737:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016246.896739:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016246.896744:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 513 00000100:00080000:0.0:1734016246.896747:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016246.896751:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016246.896762:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016246.896774:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334066688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016246.896783:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016246.896787:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016246.896789:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016246.896793:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 513 00000100:00080000:0.0:1734016246.896796:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016246.896800:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016246.896806:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016246.896816:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334066816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016246.896826:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016246.896829:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016246.896830:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016246.896834:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 513 00000100:00080000:0.0:1734016246.896836:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016246.896840:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016246.896844:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016246.896852:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334066944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016246.896861:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016246.896865:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1734016246.896865:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334066560:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016246.896866:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016246.896870:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 513 00000100:00080000:0.0:1734016246.896873:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016246.896876:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016246.896881:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016246.896889:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334067072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016246.896898:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016246.896901:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016246.896904:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016246.896908:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 513 00000100:00080000:0.0:1734016246.896910:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016246.896914:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016246.896918:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016246.896929:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334067200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016246.896937:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1734016246.896937:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734016246.896940:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016246.896942:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016246.896945:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 513 00000100:00080000:0.0:1734016246.896949:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016246.896952:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016246.896956:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016246.896959:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016246/real 1734016246] req@ffff8800a934fb80 x1818247334066560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016246.896964:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334067328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016246.896980:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334066560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016246.896991:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016246.896995:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016246.897000:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334066560:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016246.897008:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334066688:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016246.897025:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016246.897033:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016246/real 1734016246] req@ffff8800a934d500 x1818247334066688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016246.897045:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334066688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016246.897052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016246.897055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016246.897058:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334066688:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016246.897065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334066816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016246.897078:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016246.897085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016246/real 1734016246] req@ffff8800a934df80 x1818247334066816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016246.897095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334066816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016246.897103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016246.897105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016246.897108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334066816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016246.897135:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334066944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016246.897148:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016246.897155:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016246/real 1734016246] req@ffff8800a934ca80 x1818247334066944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016246.897166:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334066944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016246.897173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016246.897176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016246.897179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334066944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016246.897185:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334067072:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016246.897198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016246.897204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016246/real 1734016246] req@ffff8800a934ed80 x1818247334067072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016246.897213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334067072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016246.897220:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016246.897223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016246.897226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334067072:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016246.897232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334067200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016246.897244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016246.897250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016246/real 1734016246] req@ffff8800a934c700 x1818247334067200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016246.897260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334067200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016246.897267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016246.897270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016246.897273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334067200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016246.897279:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334067328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016246.897292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016246.897298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016246/real 1734016246] req@ffff8800a934d880 x1818247334067328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016246.897307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334067328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016246.897314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016246.897317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016246.897320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334067328:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016251.904614:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016251.904622:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016251.904627:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016251.904637:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 518 00000100:00080000:0.0:1734016251.904646:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016251.904654:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016251.904691:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334067456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016251.904732:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016251.904737:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016251.904738:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016251.904743:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 518 00000100:00080000:0.0:1734016251.904746:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016251.904750:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016251.904760:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016251.904771:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334067584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016251.904780:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016251.904783:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016251.904785:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016251.904789:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 518 00000100:00080000:0.0:1734016251.904792:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016251.904795:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016251.904800:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016251.904808:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334067712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016251.904816:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016251.904819:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016251.904821:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016251.904824:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 518 00000100:00080000:0.0:1734016251.904827:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016251.904830:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016251.904835:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016251.904844:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334067840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016251.904844:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334067456:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016251.904853:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016251.904856:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016251.904857:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016251.904861:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 518 00000100:00080000:0.0:1734016251.904863:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016251.904866:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016251.904870:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016251.904878:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334067968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016251.904886:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016251.904889:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016251.904891:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016251.904894:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 518 00000100:00080000:0.0:1734016251.904897:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016251.904900:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016251.904904:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016251.904911:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016251.904912:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334068096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016251.904919:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016251.904923:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016251.904925:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016251.904928:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 518 00000100:00080000:0.0:1734016251.904931:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1734016251.904931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016251/real 1734016251] req@ffff8800a934dc00 x1818247334067456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016251.904934:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016251.904938:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016251.904945:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334068224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016251.904951:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334067456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016251.904961:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016251.904965:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016251.904970:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334067456:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016251.904977:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334067584:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016251.904996:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016251.905003:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016251/real 1734016251] req@ffff8800a934d180 x1818247334067584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016251.905014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334067584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016251.905022:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016251.905024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016251.905028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334067584:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016251.905034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334067712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016251.905047:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016251.905053:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016251/real 1734016251] req@ffff8800a934c000 x1818247334067712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016251.905063:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334067712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016251.905071:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016251.905073:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016251.905076:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334067712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016251.905100:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334067840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016251.905113:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016251.905119:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016251/real 1734016251] req@ffff8800a934d880 x1818247334067840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016251.905136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334067840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016251.905143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016251.905145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016251.905148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334067840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016251.905153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334067968:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016251.905165:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016251.905171:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016251/real 1734016251] req@ffff8800a934c700 x1818247334067968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016251.905181:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334067968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016251.905188:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016251.905190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016251.905193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334067968:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016251.905198:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334068096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016251.905210:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016251.905216:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016251/real 1734016251] req@ffff8800a934ed80 x1818247334068096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016251.905225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334068096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016251.905232:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016251.905235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016251.905238:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334068096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016251.905243:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334068224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016251.905256:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016251.905261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016251/real 1734016251] req@ffff8800a934ca80 x1818247334068224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016251.905271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334068224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016251.905312:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016251.905315:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016251.905318:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334068224:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016256.912626:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016256.912639:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016256.912644:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016256.912655:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 523 00000100:00080000:0.0:1734016256.912664:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016256.912671:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016256.912707:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334068352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016256.912740:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016256.912744:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016256.912746:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016256.912751:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 523 00000100:00080000:0.0:1734016256.912754:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016256.912758:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016256.912768:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016256.912779:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334068480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016256.912790:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016256.912793:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016256.912795:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016256.912798:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 523 00000100:00080000:0.0:1734016256.912801:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016256.912805:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016256.912809:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016256.912818:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334068608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016256.912826:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016256.912829:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016256.912831:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016256.912834:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 523 00000100:00080000:0.0:1734016256.912836:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016256.912840:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016256.912844:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016256.912846:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334068352:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016256.912852:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334068736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016256.912862:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016256.912865:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016256.912866:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016256.912870:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 523 00000100:00080000:0.0:1734016256.912873:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016256.912876:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016256.912880:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016256.912889:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334068864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016256.912897:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016256.912900:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016256.912901:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016256.912904:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 523 00000100:00080000:0.0:1734016256.912907:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016256.912911:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:3.0:1734016256.912912:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734016256.912915:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016256.912922:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334068992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016256.912930:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016256.912933:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016256.912934:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1734016256.912934:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016256/real 1734016256] req@ffff8800a934df80 x1818247334068352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016256.912938:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 523 00000100:00080000:0.0:1734016256.912940:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016256.912944:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016256.912947:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016256.912954:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334068352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016256.912955:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334069120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016256.912964:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016256.912967:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016256.912972:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334068352:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016256.912979:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334068480:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016256.912998:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016256.913005:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016256/real 1734016256] req@ffff8800a934d500 x1818247334068480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016256.913017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334068480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016256.913024:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016256.913027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016256.913030:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334068480:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016256.913036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334068608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016256.913050:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016256.913056:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016256/real 1734016256] req@ffff8800a934fb80 x1818247334068608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016256.913066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334068608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016256.913074:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016256.913076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016256.913079:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334068608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016256.913111:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334068736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016256.913124:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016256.913130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016256/real 1734016256] req@ffff8800a934ce00 x1818247334068736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016256.913148:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334068736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016256.913155:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016256.913157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016256.913160:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334068736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016256.913166:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334068864:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016256.913177:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016256.913183:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016256/real 1734016256] req@ffff8800a934f100 x1818247334068864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016256.913230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334068864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016256.913238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016256.913240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016256.913243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334068864:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016256.913249:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334068992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016256.913265:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016256.913271:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016256/real 1734016256] req@ffff8800a934e680 x1818247334068992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016256.913282:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334068992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016256.913289:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016256.913292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016256.913295:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334068992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016256.913301:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334069120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016256.913313:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016256.913319:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016256/real 1734016256] req@ffff8800a934e300 x1818247334069120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016256.913329:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334069120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016256.913336:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016256.913339:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016256.913342:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334069120:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016261.920608:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016261.920618:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016261.920623:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016261.920634:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 528 00000100:00080000:0.0:1734016261.920643:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016261.920651:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016261.920688:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334069248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016261.920723:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016261.920728:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016261.920730:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016261.920734:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 528 00000100:00080000:0.0:1734016261.920737:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016261.920741:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016261.920751:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016261.920762:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334069376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016261.920772:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016261.920783:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016261.920784:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016261.920789:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 528 00000100:00080000:0.0:1734016261.920792:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016261.920795:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016261.920801:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016261.920810:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334069504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016261.920819:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016261.920822:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016261.920824:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016261.920827:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 528 00000100:00080000:0.0:1734016261.920830:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016261.920833:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016261.920837:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016261.920841:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334069248:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016261.920847:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334069632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016261.920856:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016261.920858:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016261.920860:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016261.920863:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 528 00000100:00080000:0.0:1734016261.920866:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016261.920869:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016261.920873:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016261.920881:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334069760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016261.920889:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016261.920892:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016261.920894:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016261.920898:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 528 00000100:00080000:0.0:1734016261.920901:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016261.920904:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016261.920908:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016261.920916:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334069888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016261.920924:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016261.920927:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016261.920928:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1734016261.920931:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334069376:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734016261.920932:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 528 00000100:00080000:0.0:1734016261.920934:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016261.920937:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016261.920941:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016261.920949:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334070016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016261.920951:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334069504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016261.920974:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334069632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016261.920988:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334069760:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016261.921002:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334069888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016261.921016:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334070016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016262.448805:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016261/real 1734016262] req@ffff8800a934e300 x1818247334069632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016262.448828:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334069632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016262.448838:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016262.448842:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016262.448846:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334069632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016262.448853:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016261/real 1734016262] req@ffff8800a934e680 x1818247334069760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016262.448864:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334069760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016262.448870:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016262.448873:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016262.448876:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334069760:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016262.448882:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016261/real 1734016262] req@ffff8800a934f100 x1818247334069888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016262.448891:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334069888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016262.448897:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016262.448900:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016262.448909:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334069888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016262.448915:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016261/real 1734016262] req@ffff8800a934ce00 x1818247334070016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016262.448942:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334070016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016262.448948:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016262.448951:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016262.448954:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334070016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016262.448959:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016261/real 1734016262] req@ffff8800a934ea00 x1818247334069248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016262.448970:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334069248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016262.448977:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016262.448979:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016262.448983:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334069248:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016262.448989:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016261/real 1734016262] req@ffff8800a934c380 x1818247334069376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016262.448999:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334069376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016262.449006:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016262.449009:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016262.449012:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334069376:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016262.449018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016261/real 1734016262] req@ffff8800a934f480 x1818247334069504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016262.449034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334069504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016262.449041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016262.449043:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016262.449046:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334069504:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016266.928614:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016266.928624:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016266.928628:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016266.928639:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 533 00000100:00080000:0.0:1734016266.928648:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016266.928656:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016266.928692:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334070144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016266.928734:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016266.928738:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016266.928740:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016266.928744:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 533 00000100:00080000:0.0:1734016266.928747:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016266.928751:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016266.928761:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016266.928772:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334070272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016266.928781:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016266.928784:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016266.928786:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016266.928790:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 533 00000100:00080000:0.0:1734016266.928793:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016266.928796:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016266.928801:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016266.928810:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334070400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016266.928818:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016266.928821:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016266.928823:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016266.928826:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 533 00000100:00080000:0.0:1734016266.928828:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016266.928832:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016266.928836:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016266.928841:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334070144:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016266.928845:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334070528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016266.928854:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016266.928857:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016266.928858:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016266.928862:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 533 00000100:00080000:0.0:1734016266.928864:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016266.928867:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016266.928871:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016266.928878:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334070656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016266.928886:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016266.928889:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016266.928891:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016266.928894:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 533 00000100:00080000:0.0:1734016266.928897:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016266.928900:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016266.928904:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016266.928908:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016266.928913:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334070784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016266.928921:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016266.928925:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016266.928927:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016266.928930:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 533 00000100:00100000:3.0:1734016266.928930:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016266/real 1734016266] req@ffff8800a934fb80 x1818247334070144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016266.928933:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016266.928936:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016266.928940:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016266.928947:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334070912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016266.928950:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334070144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016266.928960:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016266.928963:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016266.928968:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334070144:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016266.928975:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334070272:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016266.928992:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016266.928998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016266/real 1734016266] req@ffff8800a934d500 x1818247334070272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016266.929010:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334070272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016266.929017:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016266.929020:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016266.929053:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334070272:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016266.929061:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334070400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016266.929079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016266.929086:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016266/real 1734016266] req@ffff8800a934df80 x1818247334070400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016266.929097:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334070400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016266.929105:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016266.929107:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016266.929110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334070400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016266.929134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334070528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016266.929148:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016266.929154:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016266/real 1734016266] req@ffff8800a934ca80 x1818247334070528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016266.929164:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334070528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016266.929172:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016266.929174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016266.929177:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334070528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016266.929183:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334070656:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016266.929198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016266.929204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016266/real 1734016266] req@ffff8800a934ed80 x1818247334070656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016266.929214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334070656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016266.929221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016266.929224:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016266.929227:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334070656:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016266.929232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334070784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016266.929244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016266.929250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016266/real 1734016266] req@ffff8800a934c700 x1818247334070784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016266.929260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334070784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016266.929267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016266.929269:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016266.929272:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334070784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016266.929277:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334070912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016266.929289:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016266.929295:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016266/real 1734016266] req@ffff8800a934d880 x1818247334070912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016266.929310:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334070912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016266.929317:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016266.929319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016266.929322:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334070912:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016271.936577:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016271.936586:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016271.936591:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016271.936601:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 538 00000100:00080000:0.0:1734016271.936611:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016271.936619:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016271.936656:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334071040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016271.936691:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016271.936696:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016271.936697:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016271.936702:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 538 00000100:00080000:0.0:1734016271.936705:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016271.936709:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016271.936719:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016271.936730:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334071168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016271.936748:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016271.936752:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016271.936753:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016271.936757:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 538 00000100:00080000:0.0:1734016271.936760:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016271.936764:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016271.936768:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016271.936777:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334071296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016271.936786:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016271.936789:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016271.936791:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016271.936794:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 538 00000100:00080000:0.0:1734016271.936796:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016271.936800:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016271.936804:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016271.936809:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334071040:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016271.936814:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334071424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016271.936823:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016271.936826:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016271.936827:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016271.936831:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 538 00000100:00080000:0.0:1734016271.936834:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016271.936837:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016271.936841:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016271.936849:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334071552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016271.936857:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016271.936860:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016271.936861:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016271.936864:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 538 00000100:00080000:0.0:1734016271.936867:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016271.936871:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:3.0:1734016271.936874:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734016271.936875:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016271.936882:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334071680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016271.936890:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016271.936893:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016271.936894:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1734016271.936895:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016271/real 1734016271] req@ffff8800a934c000 x1818247334071040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016271.936898:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 538 00000100:00080000:0.0:1734016271.936900:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016271.936903:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016271.936907:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016271.936914:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334071808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016271.936914:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334071040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016271.936925:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016271.936928:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016271.936933:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334071040:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016271.936979:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334071168:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016271.937001:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016271.937009:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016271/real 1734016271] req@ffff8800a934d180 x1818247334071168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016271.937020:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334071168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016271.937028:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016271.937036:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016271.937040:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334071168:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016271.937046:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334071296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016271.937060:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016271.937066:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016271/real 1734016271] req@ffff8800a934dc00 x1818247334071296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016271.937076:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334071296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016271.937084:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016271.937091:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016271.937095:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334071296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016271.937119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334071424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016271.937132:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016271.937138:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016271/real 1734016271] req@ffff8800a934d880 x1818247334071424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016271.937149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334071424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016271.937156:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016271.937158:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016271.937161:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334071424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016271.937167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334071552:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016271.937182:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016271.937188:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016271/real 1734016271] req@ffff8800a934c700 x1818247334071552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016271.937199:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334071552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016271.937206:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016271.937208:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016271.937211:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334071552:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016271.937217:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334071680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016271.937228:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016271.937234:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016271/real 1734016271] req@ffff8800a934ed80 x1818247334071680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016271.937244:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334071680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016271.937251:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016271.937253:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016271.937256:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334071680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016271.937261:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334071808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016271.937273:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016271.937279:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016271/real 1734016271] req@ffff8800a934ca80 x1818247334071808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016271.937296:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334071808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016271.937303:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016271.937305:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016271.937308:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334071808:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016276.944609:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016276.944618:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016276.944623:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016276.944634:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 543 00000100:00080000:0.0:1734016276.944642:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016276.944650:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016276.944686:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334071936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016276.944720:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016276.944724:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016276.944726:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016276.944731:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 543 00000100:00080000:0.0:1734016276.944734:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016276.944738:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016276.944749:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016276.944759:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334072064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016276.944769:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016276.944772:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016276.944774:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016276.944778:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 543 00000100:00080000:0.0:1734016276.944781:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016276.944784:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016276.944790:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016276.944799:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334072192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016276.944808:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016276.944811:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016276.944813:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016276.944816:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 543 00000100:00080000:0.0:1734016276.944818:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016276.944822:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016276.944826:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016276.944826:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334071936:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016276.944833:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334072320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016276.944842:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016276.944845:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016276.944846:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016276.944850:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 543 00000100:00080000:0.0:1734016276.944852:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016276.944891:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016276.944896:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016276.944906:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334072448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016276.944915:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016276.944918:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016276.944920:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1734016276.944923:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734016276.944924:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 543 00000100:00080000:0.0:1734016276.944927:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016276.944931:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016276.944935:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016276.944943:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334072576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016276.944943:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016276/real 1734016276] req@ffff8800a934df80 x1818247334071936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016276.944951:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016276.944954:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016276.944956:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016276.944959:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 543 00000100:00080000:0.0:1734016276.944962:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1734016276.944962:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334071936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016276.944966:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016276.944970:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1734016276.944973:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1734016276.944977:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334072704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016276.944977:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016276.944981:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334071936:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016276.944989:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334072064:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016276.945005:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016276.945012:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016276/real 1734016276] req@ffff8800a934d500 x1818247334072064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016276.945023:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334072064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016276.945030:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016276.945034:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016276.945037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334072064:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016276.945044:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334072192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016276.945057:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016276.945063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016276/real 1734016276] req@ffff8800a934fb80 x1818247334072192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016276.945074:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334072192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016276.945081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016276.945084:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016276.945087:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334072192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016276.945111:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334072320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016276.945124:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016276.945130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016276/real 1734016276] req@ffff8800a934f480 x1818247334072320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016276.945140:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334072320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016276.945148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016276.945150:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016276.945153:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334072320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016276.945158:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334072448:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016276.945174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016276.945180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016276/real 1734016276] req@ffff8800a934c380 x1818247334072448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016276.945191:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334072448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016276.945198:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016276.945200:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016276.945203:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334072448:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016276.945208:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334072576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016276.945219:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016276.945225:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016276/real 1734016276] req@ffff8800a934ea00 x1818247334072576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016276.945236:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334072576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016276.945243:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016276.945245:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016276.945248:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334072576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016276.945253:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334072704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016276.945265:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016276.945271:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016276/real 1734016276] req@ffff8800a934ce00 x1818247334072704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016276.945287:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334072704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016276.945294:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016276.945296:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016276.945299:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334072704:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016281.952613:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016281.952622:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016281.952627:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016281.952637:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 548 00000100:00080000:0.0:1734016281.952646:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016281.952654:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016281.952690:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334072832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016281.952730:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016281.952735:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016281.952736:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016281.952741:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 548 00000100:00080000:0.0:1734016281.952744:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016281.952748:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016281.952758:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016281.952769:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334072960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016281.952818:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016281.952822:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016281.952823:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016281.952828:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 548 00000100:00080000:0.0:1734016281.952831:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016281.952835:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016281.952840:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016281.952850:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334073088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016281.952857:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334072832:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016281.952860:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016281.952863:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016281.952865:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016281.952868:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 548 00000100:00080000:0.0:1734016281.952871:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016281.952875:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016281.952879:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016281.952889:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334073216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016281.952897:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016281.952900:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016281.952901:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016281.952904:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 548 00000100:00080000:0.0:1734016281.952907:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016281.952910:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016281.952914:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1734016281.952920:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016281.952922:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334073344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016281.952931:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016281.952934:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016281.952935:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016281.952938:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 548 00000100:00080000:0.0:1734016281.952941:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1734016281.952942:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016281/real 1734016281] req@ffff8800a934f100 x1818247334072832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016281.952945:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016281.952948:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016281.952956:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334073472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016281.952961:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334072832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016281.952964:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016281.952968:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016281.952969:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016281.952972:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016281.952973:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 548 00000100:00080000:0.0:1734016281.952975:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016281.952975:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734016281.952978:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:3.0:1734016281.952980:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334072832:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734016281.952982:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016281.952987:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334072960:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016281.952990:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334073600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:3.0:1734016281.953004:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016281.953011:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016281/real 1734016281] req@ffff8800a934e680 x1818247334072960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016281.953022:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334072960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016281.953029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016281.953033:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016281.953036:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334072960:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016281.953059:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334073088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016281.953073:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016281.953079:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016281/real 1734016281] req@ffff8800a934e300 x1818247334073088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016281.953090:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334073088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016281.953098:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016281.953100:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016281.953103:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334073088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016281.953109:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334073216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016281.953120:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016281.953126:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016281/real 1734016281] req@ffff8800a934ce00 x1818247334073216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016281.953136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334073216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016281.953143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016281.953145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016281.953148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334073216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016281.953154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334073344:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016281.953170:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016281.953176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016281/real 1734016281] req@ffff8800a934ea00 x1818247334073344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016281.953187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334073344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016281.953194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016281.953196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016281.953199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334073344:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016281.953204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334073472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016281.953216:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016281.953222:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016281/real 1734016281] req@ffff8800a934c380 x1818247334073472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016281.953232:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334073472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016281.953238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016281.953241:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016281.953244:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334073472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016281.953249:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334073600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016281.953261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016281.953266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016281/real 1734016281] req@ffff8800a934f480 x1818247334073600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016281.953282:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334073600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016281.953289:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016281.953291:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016281.953294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334073600:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016286.960610:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016286.960623:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016286.960628:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016286.960643:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 553 00000100:00080000:0.0:1734016286.960651:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016286.960659:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016286.960732:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334073728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016286.960764:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016286.960768:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016286.960770:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016286.960775:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 553 00000100:00080000:0.0:1734016286.960778:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016286.960782:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016286.960792:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016286.960804:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334073856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016286.960815:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016286.960818:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016286.960820:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016286.960824:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 553 00000100:00080000:0.0:1734016286.960827:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016286.960830:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016286.960835:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016286.960844:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334073984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016286.960852:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016286.960855:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016286.960857:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016286.960860:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 553 00000100:00080000:0.0:1734016286.960863:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016286.960866:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016286.960870:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016286.960877:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334073728:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016286.960878:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334074112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016286.960887:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016286.960890:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016286.960892:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016286.960895:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 553 00000100:00080000:0.0:1734016286.960898:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016286.960901:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016286.960905:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016286.960913:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334074240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016286.960921:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016286.960923:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016286.960925:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016286.960928:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 553 00000100:00080000:0.0:1734016286.960931:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016286.960934:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016286.960938:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016286.960942:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016286.960948:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334074368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016286.960956:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016286.960958:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016286.960960:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016286.960963:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 553 00000100:00100000:3.0:1734016286.960964:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016286/real 1734016286] req@ffff8800a934fb80 x1818247334073728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016286.960966:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016286.960969:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016286.960973:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016286.960980:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334074496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016286.960984:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334073728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016286.960995:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016286.960998:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016286.961003:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334073728:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016286.961010:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334073856:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016286.961026:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016286.961032:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016286/real 1734016286] req@ffff8800a934d500 x1818247334073856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016286.961044:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334073856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016286.961051:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016286.961055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016286.961058:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334073856:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016286.961064:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334073984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016286.961077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016286.961083:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016286/real 1734016286] req@ffff8800a934df80 x1818247334073984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016286.961093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334073984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016286.961101:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016286.961103:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016286.961107:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334073984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016286.961130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334074112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016286.961143:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016286.961149:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016286/real 1734016286] req@ffff8800a934ca80 x1818247334074112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016286.961159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334074112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016286.961167:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016286.961169:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016286.961172:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334074112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016286.961177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334074240:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016286.961193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016286.961199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016286/real 1734016286] req@ffff8800a934ed80 x1818247334074240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016286.961210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334074240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016286.961217:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016286.961219:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016286.961222:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334074240:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016286.961227:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334074368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016286.961238:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016286.961244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016286/real 1734016286] req@ffff8800a934c700 x1818247334074368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016286.961255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334074368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016286.961262:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016286.961264:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016286.961267:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334074368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016286.961273:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334074496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016286.961285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016286.961291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016286/real 1734016286] req@ffff8800a934d880 x1818247334074496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016286.961307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334074496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016286.961315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016286.961317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016286.961320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334074496:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016291.968631:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016291.968642:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016291.968647:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016291.968657:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 558 00000100:00080000:0.0:1734016291.968666:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016291.968674:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016291.968711:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334074624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016291.968745:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016291.968749:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016291.968751:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016291.968755:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 558 00000100:00080000:0.0:1734016291.968759:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016291.968763:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016291.968773:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016291.968784:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334074752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016291.968794:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016291.968797:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016291.968798:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016291.968803:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 558 00000100:00080000:0.0:1734016291.968805:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016291.968809:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016291.968814:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016291.968823:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334074880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016291.968832:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016291.968835:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016291.968837:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016291.968840:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 558 00000100:00080000:0.0:1734016291.968842:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016291.968845:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016291.968849:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016291.968857:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334074624:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016291.968859:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334075008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016291.968868:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016291.968871:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016291.968872:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016291.968875:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 558 00000100:00080000:0.0:1734016291.968878:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016291.968881:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016291.968885:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016291.968893:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334075136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016291.968902:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016291.968905:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016291.968907:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016291.968911:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 558 00000100:00080000:0.0:1734016291.968913:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016291.968917:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016291.968921:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016291.968921:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016291.968928:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334075264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016291.968936:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016291.968939:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016291.968941:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1734016291.968942:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016291/real 1734016291] req@ffff8800a934dc00 x1818247334074624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016291.968944:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 558 00000100:00080000:0.0:1734016291.968947:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016291.968950:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016291.968953:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016291.968960:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334075392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016291.968961:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334074624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016291.968972:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016291.968975:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016291.968980:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334074624:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016291.968987:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334074752:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016291.969002:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016291.969009:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016291/real 1734016291] req@ffff8800a934d180 x1818247334074752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016291.969020:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334074752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016291.969028:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016291.969031:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016291.969034:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334074752:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016291.969041:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334074880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016291.969054:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016291.969061:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016291/real 1734016291] req@ffff8800a934c000 x1818247334074880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016291.969070:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334074880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016291.969078:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016291.969081:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016291.969084:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334074880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016291.969107:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334075008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016291.969120:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016291.969126:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016291/real 1734016291] req@ffff8800a934d880 x1818247334075008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016291.969136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334075008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016291.969144:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016291.969146:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016291.969149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334075008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016291.969154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334075136:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016291.969169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016291.969176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016291/real 1734016291] req@ffff8800a934c700 x1818247334075136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016291.969186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334075136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016291.969193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016291.969195:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016291.969198:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334075136:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016291.969204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334075264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016291.969215:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016291.969221:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016291/real 1734016291] req@ffff8800a934ed80 x1818247334075264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016291.969231:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334075264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016291.969238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016291.969240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016291.969242:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334075264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016291.969248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334075392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016291.969260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016291.969266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016291/real 1734016291] req@ffff8800a934ca80 x1818247334075392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016291.969282:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334075392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016291.969290:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016291.969292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016291.969295:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334075392:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016296.976616:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016296.976630:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016296.976635:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016296.976645:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 563 00000100:00080000:0.0:1734016296.976654:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016296.976662:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016296.976698:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334075520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016296.976739:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016296.976743:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016296.976745:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016296.976749:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 563 00000100:00080000:0.0:1734016296.976753:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016296.976756:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016296.976766:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016296.976778:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334075648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016296.976787:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016296.976790:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016296.976792:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016296.976796:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 563 00000100:00080000:0.0:1734016296.976798:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016296.976802:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016296.976806:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016296.976815:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334075776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016296.976824:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016296.976827:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016296.976828:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016296.976831:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 563 00000100:00080000:0.0:1734016296.976834:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016296.976837:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:3.0:1734016296.976841:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334075520:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734016296.976842:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016296.976850:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334075904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016296.976859:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016296.976862:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016296.976863:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016296.976867:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 563 00000100:00080000:0.0:1734016296.976869:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016296.976872:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016296.976876:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016296.976885:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334076032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016296.976893:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016296.976896:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016296.976897:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016296.976901:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 563 00000100:00080000:0.0:1734016296.976903:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016296.976906:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016296.976910:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016296.976918:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334076160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016296.976926:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016296.976929:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016296.976931:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1734016296.976931:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334075648:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734016296.976935:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 563 00000100:00080000:0.0:1734016296.976938:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016296.976941:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016296.976945:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016296.976951:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334075776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016296.976952:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334076288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016296.976974:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334075904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016296.976989:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334076032:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016296.977003:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334076160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016296.977018:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334076288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016299.982966:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016296/real 1734016299] req@ffff8800a934f480 x1818247334075904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016299.983005:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334075904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016299.983022:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016299.983027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016299.983034:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334075904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016299.983047:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016296/real 1734016299] req@ffff8800a934c380 x1818247334076032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016299.983066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334076032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016299.983077:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016299.983081:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016299.983087:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334076032:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016299.983097:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016296/real 1734016299] req@ffff8800a934ea00 x1818247334076160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016299.983114:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334076160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016299.983127:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016299.983130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016299.983136:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334076160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016299.983146:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016296/real 1734016299] req@ffff8800a934ce00 x1818247334076288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016299.983164:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334076288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016299.983176:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016299.983180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016299.983186:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334076288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016299.983200:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016296/real 1734016299] req@ffff8800a934df80 x1818247334075520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016299.983218:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334075520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016299.983230:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016299.983234:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016299.983239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334075520:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016299.983249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016296/real 1734016299] req@ffff8800a934d500 x1818247334075648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016299.983266:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334075648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016299.983279:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016299.983284:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016299.983289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334075648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016299.983300:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016296/real 1734016299] req@ffff8800a934fb80 x1818247334075776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016299.983319:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334075776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016299.983332:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016299.983336:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016299.983342:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334075776:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016301.984592:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016301.984601:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016301.984605:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016301.984616:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 568 00000100:00080000:0.0:1734016301.984626:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016301.984634:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016301.984670:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334076416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016301.984704:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016301.984708:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016301.984710:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016301.984714:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 568 00000100:00080000:0.0:1734016301.984717:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016301.984721:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016301.984731:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016301.984741:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334076544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016301.984752:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016301.984755:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016301.984757:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016301.984760:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 568 00000100:00080000:0.0:1734016301.984763:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016301.984766:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016301.984771:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016301.984779:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334076672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016301.984788:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016301.984791:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016301.984792:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016301.984796:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 568 00000100:00080000:0.0:1734016301.984798:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016301.984801:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016301.984805:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016301.984811:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334076416:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016301.984814:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334076800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016301.984824:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016301.984826:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016301.984828:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016301.984832:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 568 00000100:00080000:0.0:1734016301.984835:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016301.984838:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016301.984842:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016301.984850:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334076928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016301.984857:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016301.984860:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016301.984862:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016301.984865:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 568 00000100:00080000:0.0:1734016301.984868:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016301.984871:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016301.984875:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016301.984876:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016301.984882:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334077056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016301.984890:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016301.984893:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016301.984894:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1734016301.984895:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016301/real 1734016301] req@ffff8800a934e300 x1818247334076416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016301.984898:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 568 00000100:00080000:0.0:1734016301.984900:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016301.984903:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016301.984907:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016301.984914:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334076416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016301.984915:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334077184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016301.984925:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016301.984928:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016301.984933:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334076416:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016301.984940:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334076544:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016301.984956:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016301.984963:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016301/real 1734016301] req@ffff8800a934e680 x1818247334076544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016301.984974:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334076544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016301.984982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016301.984984:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016301.984988:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334076544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016301.984993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334076672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016301.985007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016301.985013:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016301/real 1734016301] req@ffff8800a934f100 x1818247334076672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016301.985023:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334076672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016301.985031:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016301.985034:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016301.985037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334076672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016301.985061:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334076800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016301.985073:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016301.985079:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016301/real 1734016301] req@ffff8800a934fb80 x1818247334076800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016301.985090:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334076800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016301.985097:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016301.985099:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016301.985102:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334076800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016301.985108:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334076928:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016301.985120:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016301.985126:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016301/real 1734016301] req@ffff8800a934d500 x1818247334076928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016301.985136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334076928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016301.985143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016301.985145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016301.985148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334076928:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016301.985154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334077056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016301.985166:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016301.985173:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016301/real 1734016301] req@ffff8800a934df80 x1818247334077056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016301.985182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334077056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016301.985189:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016301.985191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016301.985194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334077056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016301.985200:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334077184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016301.985211:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016301.985217:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016301/real 1734016301] req@ffff8800a934ce00 x1818247334077184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016301.985227:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334077184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016301.985234:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016301.985236:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016301.985239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334077184:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016306.992596:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016306.992606:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016306.992611:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016306.992622:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 573 00000100:00080000:0.0:1734016306.992632:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016306.992640:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016306.992677:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334077312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016306.992713:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016306.992717:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016306.992718:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016306.992723:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 573 00000100:00080000:0.0:1734016306.992726:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016306.992730:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016306.992740:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016306.992751:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334077440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016306.992760:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016306.992763:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016306.992765:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016306.992769:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 573 00000100:00080000:0.0:1734016306.992772:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016306.992775:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016306.992781:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016306.992790:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334077568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016306.992799:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016306.992803:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016306.992804:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016306.992807:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 573 00000100:00080000:0.0:1734016306.992810:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016306.992813:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016306.992817:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016306.992824:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334077312:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016306.992825:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334077696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016306.992834:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016306.992837:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016306.992838:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016306.992842:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 573 00000100:00080000:0.0:1734016306.992844:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016306.992847:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016306.992851:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016306.992859:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334077824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016306.992867:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016306.992870:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016306.992872:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016306.992876:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 573 00000100:00080000:0.0:1734016306.992879:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016306.992882:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016306.992886:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016306.992888:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016306.992896:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334077952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016306.992903:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016306.992907:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016306.992908:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1734016306.992909:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016306/real 1734016306] req@ffff8800a934ea00 x1818247334077312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016306.992912:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 573 00000100:00080000:0.0:1734016306.992914:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016306.992917:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016306.992921:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016306.992928:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334078080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016306.992929:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334077312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016306.992940:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016306.992943:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016306.992947:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334077312:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016306.992955:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334077440:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016306.992971:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016306.992978:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016306/real 1734016306] req@ffff8800a934c380 x1818247334077440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016306.992990:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334077440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016306.992997:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016306.992999:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016306.993003:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334077440:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016306.993008:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334077568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016306.993021:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016306.993027:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016306/real 1734016306] req@ffff8800a934f480 x1818247334077568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016306.993037:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334077568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016306.993045:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016306.993047:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016306.993050:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334077568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016306.993074:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334077696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016306.993087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016306.993093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016306/real 1734016306] req@ffff8800a934ca80 x1818247334077696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016306.993104:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334077696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016306.993111:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016306.993113:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016306.993116:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334077696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016306.993121:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334077824:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016306.993133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016306.993139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016306/real 1734016306] req@ffff8800a934ed80 x1818247334077824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016306.993148:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334077824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016306.993155:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016306.993157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016306.993160:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334077824:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016306.993166:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334077952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016306.993181:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016306.993187:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016306/real 1734016306] req@ffff8800a934c700 x1818247334077952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016306.993196:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334077952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016306.993203:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016306.993205:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016306.993208:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334077952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016306.993213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334078080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016306.993225:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016306.993231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016306/real 1734016306] req@ffff8800a934d880 x1818247334078080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016306.993241:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334078080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016306.993248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016306.993250:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016306.993253:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334078080:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016312.000582:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016312.000593:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016312.000598:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016312.000611:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 578 00000100:00080000:0.0:1734016312.000623:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016312.000632:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016312.000679:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334078208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016312.000726:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016312.000731:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016312.000733:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016312.000738:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 578 00000100:00080000:0.0:1734016312.000741:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016312.000745:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016312.000756:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016312.000768:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334078336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016312.000777:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016312.000780:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016312.000782:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016312.000786:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 578 00000100:00080000:0.0:1734016312.000789:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016312.000793:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016312.000798:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016312.000807:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334078464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016312.000816:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016312.000819:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016312.000820:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016312.000824:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 578 00000100:00080000:0.0:1734016312.000826:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016312.000831:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016312.000835:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016312.000847:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334078592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016312.000848:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334078208:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016312.000857:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016312.000860:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016312.000862:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016312.000865:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 578 00000100:00080000:0.0:1734016312.000868:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016312.000871:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016312.000875:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016312.000884:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334078720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016312.000892:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016312.000895:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016312.000896:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016312.000900:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 578 00000100:00080000:0.0:1734016312.000903:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016312.000907:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016312.000912:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016312.000921:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334078848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:3.0:1734016312.000921:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734016312.000929:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016312.000933:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016312.000934:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016312.000938:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 578 00000100:00080000:0.0:1734016312.000942:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1734016312.000943:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016312/real 1734016312] req@ffff8800a934c000 x1818247334078208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016312.000945:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016312.000950:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016312.000958:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334078976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016312.000971:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334078208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016312.000982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016312.000985:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016312.000990:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334078208:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016312.000998:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334078336:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016312.001015:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016312.001023:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016312/real 1734016312] req@ffff8800a934d180 x1818247334078336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016312.001034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334078336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016312.001041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016312.001044:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016312.001047:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334078336:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016312.001053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334078464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016312.001067:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016312.001073:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016312/real 1734016312] req@ffff8800a934dc00 x1818247334078464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016312.001084:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334078464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016312.001091:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016312.001094:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016312.001097:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334078464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016312.001124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334078592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016312.001138:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016312.001144:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016312/real 1734016312] req@ffff8800a934d880 x1818247334078592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016312.001155:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334078592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016312.001162:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016312.001165:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016312.001168:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334078592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016312.001174:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334078720:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016312.001186:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016312.001192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016312/real 1734016312] req@ffff8800a934c700 x1818247334078720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016312.001202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334078720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016312.001209:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016312.001212:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016312.001214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334078720:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016312.001220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334078848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016312.001235:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016312.001241:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016312/real 1734016312] req@ffff8800a934ed80 x1818247334078848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016312.001251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334078848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016312.001258:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016312.001291:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016312.001295:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334078848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016312.001302:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334078976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016312.001319:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016312.001326:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016312/real 1734016312] req@ffff8800a934ca80 x1818247334078976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016312.001336:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334078976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016312.001344:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016312.001346:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016312.001349:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334078976:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016317.008602:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016317.008611:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016317.008616:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016317.008626:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 583 00000100:00080000:0.0:1734016317.008635:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016317.008643:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016317.008679:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334079104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016317.008715:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016317.008721:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016317.008723:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016317.008730:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 583 00000100:00080000:0.0:1734016317.008735:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016317.008741:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016317.008756:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016317.008773:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334079232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016317.008781:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334079104:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016317.008790:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016317.008794:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016317.008796:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016317.008802:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 583 00000100:00080000:0.0:1734016317.008805:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016317.008811:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016317.008818:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016317.008830:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334079360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016317.008842:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016317.008847:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016317.008849:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1734016317.008851:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734016317.008854:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 583 00000100:00080000:0.0:1734016317.008857:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016317.008862:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016317.008868:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016317.008875:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016317/real 1734016317] req@ffff8800a934f480 x1818247334079104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016317.008880:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334079488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016317.008894:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016317.008899:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016317.008901:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016317.008906:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 583 00000100:00080000:0.0:1734016317.008910:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1734016317.008910:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334079104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016317.008915:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016317.008922:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1734016317.008924:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016317.008928:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016317.008933:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334079616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016317.008935:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334079104:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016317.008946:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016317.008950:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016317.008952:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016317.008957:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 583 00000100:00080000:0.0:1734016317.008961:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1734016317.008962:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334079232:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734016317.008966:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016317.008972:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016317.008985:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334079744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:3.0:1734016317.008987:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016317.008998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016317/real 1734016317] req@ffff8800a934c380 x1818247334079232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016317.008999:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016317.009004:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016317.009007:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016317.009012:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 583 00000100:00100000:3.0:1734016317.009015:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334079232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016317.009016:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016317.009022:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016317.009028:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1734016317.009028:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016317.009032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016317.009037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334079232:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016317.009040:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334079872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016317.009047:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334079360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016317.009068:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016317.009079:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016317/real 1734016317] req@ffff8800a934ea00 x1818247334079360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016317.009096:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334079360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016317.009109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016317.009113:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016317.009119:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334079360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016317.009128:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334079488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016317.009153:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016317.009165:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016317/real 1734016317] req@ffff8800a934ce00 x1818247334079488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016317.009211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334079488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016317.009220:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016317.009223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016317.009226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334079488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016317.009233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334079616:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016317.009253:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016317.009260:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016317/real 1734016317] req@ffff8800a934df80 x1818247334079616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016317.009271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334079616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016317.009279:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016317.009281:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016317.009284:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334079616:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016317.009302:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334079744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016317.009318:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016317.009324:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016317/real 1734016317] req@ffff8800a934d500 x1818247334079744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016317.009334:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334079744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016317.009341:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016317.009344:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016317.009347:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334079744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016317.009352:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334079872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016317.009364:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016317.009370:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016317/real 1734016317] req@ffff8800a934fb80 x1818247334079872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016317.009379:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334079872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016317.009386:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016317.009388:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016317.009391:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334079872:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016322.016606:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016322.016615:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016322.016620:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016322.016630:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 588 00000100:00080000:0.0:1734016322.016639:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016322.016647:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016322.016682:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334080000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016322.016717:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016322.016721:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016322.016723:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016322.016728:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 588 00000100:00080000:0.0:1734016322.016731:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016322.016735:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016322.016745:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016322.016756:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334080128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016322.016765:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016322.016769:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016322.016770:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016322.016774:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 588 00000100:00080000:0.0:1734016322.016777:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016322.016780:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016322.016786:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016322.016795:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334080256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016322.016804:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016322.016808:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016322.016809:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016322.016812:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 588 00000100:00080000:0.0:1734016322.016815:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016322.016818:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016322.016822:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016322.016832:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334080384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016322.016834:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334080000:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734016322.016841:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016322.016844:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016322.016845:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016322.016849:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 588 00000100:00080000:0.0:1734016322.016851:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016322.016854:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016322.016858:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016322.016866:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334080512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016322.016874:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016322.016877:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016322.016879:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016322.016882:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 588 00000100:00080000:0.0:1734016322.016885:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016322.016889:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016322.016893:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016322.016900:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334080640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:3.0:1734016322.016905:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734016322.016908:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016322.016911:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016322.016912:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016322.016916:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 588 00000100:00080000:0.0:1734016322.016918:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016322.016921:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016322.016925:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016322.016926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016322/real 1734016322] req@ffff8800a934f100 x1818247334080000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016322.016933:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334080768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016322.016953:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334080000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016322.016964:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016322.016967:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016322.016972:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334080000:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016322.016979:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334080128:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016322.016995:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016322.017002:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016322/real 1734016322] req@ffff8800a934e680 x1818247334080128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016322.017013:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334080128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016322.017020:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016322.017022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016322.017026:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334080128:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016322.017032:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334080256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016322.017044:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016322.017050:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016322/real 1734016322] req@ffff8800a934e300 x1818247334080256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016322.017060:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334080256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016322.017068:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016322.017072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016322.017075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334080256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016322.017128:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334080384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016322.017145:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016322.017153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016322/real 1734016322] req@ffff8800a934fb80 x1818247334080384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016322.017163:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334080384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016322.017171:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016322.017173:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016322.017176:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334080384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016322.017181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334080512:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016322.017193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016322.017200:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016322/real 1734016322] req@ffff8800a934d500 x1818247334080512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016322.017210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334080512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016322.017218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016322.017220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016322.017223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334080512:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016322.017228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334080640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016322.017243:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016322.017249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016322/real 1734016322] req@ffff8800a934df80 x1818247334080640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016322.017260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334080640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016322.017266:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016322.017269:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016322.017272:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334080640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016322.017277:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334080768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016322.017289:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016322.017295:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016322/real 1734016322] req@ffff8800a934ce00 x1818247334080768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016322.017305:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334080768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016322.017311:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016322.017313:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016322.017316:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334080768:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016327.024607:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016327.024620:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016327.024625:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016327.024635:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 593 00000100:00080000:0.0:1734016327.024643:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016327.024651:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016327.024689:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334080896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016327.024730:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016327.024734:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016327.024736:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016327.024740:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 593 00000100:00080000:0.0:1734016327.024744:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016327.024748:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016327.024758:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016327.024769:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334081024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016327.024779:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016327.024782:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016327.024784:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016327.024788:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 593 00000100:00080000:0.0:1734016327.024790:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016327.024794:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016327.024799:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016327.024807:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334081152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016327.024816:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016327.024819:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016327.024820:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016327.024824:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 593 00000100:00080000:0.0:1734016327.024826:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016327.024830:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016327.024834:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016327.024839:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334080896:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016327.024842:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334081280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016327.024851:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016327.024854:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016327.024856:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016327.024859:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 593 00000100:00080000:0.0:1734016327.024861:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016327.024865:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016327.024868:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016327.024875:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334081408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016327.024883:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016327.024886:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016327.024888:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016327.024891:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 593 00000100:00080000:0.0:1734016327.024894:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016327.024897:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016327.024901:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016327.024903:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016327.024910:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334081536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016327.024918:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016327.024922:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016327.024924:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1734016327.024925:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016327/real 1734016327] req@ffff8800a934ea00 x1818247334080896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016327.024927:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 593 00000100:00080000:0.0:1734016327.024930:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016327.024933:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016327.024937:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016327.024944:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334081664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016327.024952:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334080896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016327.024962:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016327.024965:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016327.024970:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334080896:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016327.024978:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334081024:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016327.024993:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016327.025000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016327/real 1734016327] req@ffff8800a934c380 x1818247334081024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016327.025053:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334081024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016327.025061:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016327.025064:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016327.025068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334081024:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016327.025074:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334081152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016327.025092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016327.025100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016327/real 1734016327] req@ffff8800a934f480 x1818247334081152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016327.025110:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334081152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016327.025118:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016327.025121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016327.025124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334081152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016327.025147:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334081280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016327.025160:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016327.025166:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016327/real 1734016327] req@ffff8800a934ca80 x1818247334081280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016327.025177:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334081280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016327.025184:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016327.025186:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016327.025189:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334081280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016327.025194:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334081408:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016327.025206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016327.025212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016327/real 1734016327] req@ffff8800a934ed80 x1818247334081408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016327.025222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334081408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016327.025229:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016327.025231:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016327.025234:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334081408:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016327.025240:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334081536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016327.025255:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016327.025261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016327/real 1734016327] req@ffff8800a934c700 x1818247334081536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016327.025272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334081536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016327.025278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016327.025281:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016327.025283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334081536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016327.025289:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334081664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016327.025301:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016327.025307:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016327/real 1734016327] req@ffff8800a934d880 x1818247334081664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016327.025317:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334081664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016327.025323:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016327.025325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016327.025328:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334081664:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016332.032588:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016332.032599:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016332.032604:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016332.032618:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 598 00000100:00080000:0.0:1734016332.032628:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016332.032638:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016332.032687:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334081792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016332.032726:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016332.032730:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016332.032732:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016332.032737:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 598 00000100:00080000:0.0:1734016332.032740:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016332.032744:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016332.032755:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016332.032766:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334081920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016332.032777:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016332.032781:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016332.032783:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016332.032787:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 598 00000100:00080000:0.0:1734016332.032789:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016332.032793:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016332.032798:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734016332.032806:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334081792:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016332.032808:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334082048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016332.032817:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016332.032821:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016332.032822:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016332.032826:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 598 00000100:00080000:0.0:1734016332.032828:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016332.032832:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016332.032836:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016332.032848:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334082176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016332.032857:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016332.032860:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016332.032861:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016332.032865:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 598 00000100:00080000:0.0:1734016332.032869:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016332.032873:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016332.032877:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016332.032886:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334082304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016332.032893:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016332.032896:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1734016332.032897:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734016332.032898:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016332.032902:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 598 00000100:00080000:0.0:1734016332.032906:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016332.032910:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016332.032914:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016332.032937:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016332/real 1734016332] req@ffff8800a934dc00 x1818247334081792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016332.032967:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334081792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016332.032985:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016332.032989:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016332.032994:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334081792:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016332.032997:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334082432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016332.033003:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334081920:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734016332.033008:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016332.033011:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016332.033013:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016332.033017:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 598 00000100:00080000:0.0:1734016332.033021:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016332.033025:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016332.033031:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016332.033032:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016332.033040:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016332/real 1734016332] req@ffff8800a934d180 x1818247334081920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016332.033042:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334082560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016332.033059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334081920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016332.033068:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016332.033071:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016332.033075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334081920:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016332.033110:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334082048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016332.033126:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016332.033141:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016332/real 1734016332] req@ffff8800a934c000 x1818247334082048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016332.033153:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334082048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016332.033161:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016332.033171:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016332.033174:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334082048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016332.033182:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334082176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016332.033196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016332.033212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016332/real 1734016332] req@ffff8800a934d880 x1818247334082176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016332.033222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334082176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016332.033229:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016332.033232:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016332.033235:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334082176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016332.033247:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334082304:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016332.033261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016332.033268:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016332/real 1734016332] req@ffff8800a934c700 x1818247334082304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016332.033286:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334082304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016332.033293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016332.033296:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016332.033299:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334082304:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016332.033306:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334082432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016332.033330:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016332.033336:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016332/real 1734016332] req@ffff8800a934ed80 x1818247334082432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016332.033347:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334082432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016332.033361:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016332.033364:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016332.033367:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334082432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016332.033373:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334082560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016332.033394:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016332.033401:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016332/real 1734016332] req@ffff8800a934ca80 x1818247334082560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016332.033411:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334082560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016332.033418:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016332.033427:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016332.033430:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334082560:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016337.040601:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016337.040610:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016337.040615:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016337.040625:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 603 00000100:00080000:0.0:1734016337.040634:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016337.040643:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016337.040679:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334082688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016337.040713:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016337.040717:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016337.040719:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016337.040723:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 603 00000100:00080000:0.0:1734016337.040726:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016337.040730:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016337.040741:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016337.040752:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334082816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016337.040761:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016337.040764:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016337.040766:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016337.040770:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 603 00000100:00080000:0.0:1734016337.040773:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016337.040776:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016337.040782:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016337.040791:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334082944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016337.040800:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016337.040803:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016337.040804:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016337.040808:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 603 00000100:00080000:0.0:1734016337.040810:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016337.040813:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016337.040817:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016337.040823:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334082688:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016337.040825:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334083072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016337.040864:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016337.040868:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016337.040870:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016337.040873:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 603 00000100:00080000:0.0:1734016337.040876:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016337.040879:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016337.040884:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016337.040893:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334083200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016337.040901:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016337.040904:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016337.040906:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016337.040910:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 603 00000100:00080000:0.0:1734016337.040913:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016337.040916:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016337.040920:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016337.040922:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016337.040929:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334083328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016337.040938:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016337.040941:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016337.040942:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1734016337.040944:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016337/real 1734016337] req@ffff8800a934f480 x1818247334082688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016337.040946:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 603 00000100:00080000:0.0:1734016337.040948:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016337.040951:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016337.040955:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016337.040963:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334083456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016337.040972:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334082688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016337.040982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016337.040985:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016337.040990:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334082688:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016337.040997:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334082816:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016337.041014:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016337.041020:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016337/real 1734016337] req@ffff8800a934c380 x1818247334082816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016337.041031:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334082816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016337.041038:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016337.041041:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016337.041044:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334082816:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016337.041050:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334082944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016337.041063:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016337.041069:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016337/real 1734016337] req@ffff8800a934ea00 x1818247334082944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016337.041079:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334082944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016337.041086:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016337.041089:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016337.041092:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334082944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016337.041118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334083072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016337.041131:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016337.041137:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016337/real 1734016337] req@ffff8800a934ce00 x1818247334083072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016337.041147:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334083072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016337.041155:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016337.041157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016337.041160:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334083072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016337.041165:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334083200:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016337.041177:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016337.041183:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016337/real 1734016337] req@ffff8800a934df80 x1818247334083200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016337.041192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334083200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016337.041199:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016337.041201:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016337.041204:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334083200:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016337.041210:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334083328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016337.041225:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016337.041231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016337/real 1734016337] req@ffff8800a934d500 x1818247334083328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016337.041242:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334083328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016337.041248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016337.041250:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016337.041253:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334083328:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016337.041259:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334083456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016337.041271:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016337.041277:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016337/real 1734016337] req@ffff8800a934fb80 x1818247334083456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016337.041286:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334083456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016337.041293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016337.041295:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016337.041298:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334083456:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016342.048582:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016342.048592:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016342.048596:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016342.048608:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 608 00000100:00080000:0.0:1734016342.048616:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016342.048623:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016342.048658:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334083584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016342.048700:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016342.048704:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016342.048706:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016342.048710:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 608 00000100:00080000:0.0:1734016342.048713:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016342.048717:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016342.048727:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016342.048738:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334083712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016342.048747:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016342.048774:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016342.048776:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016342.048781:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 608 00000100:00080000:0.0:1734016342.048784:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016342.048787:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016342.048792:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016342.048802:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334083840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016342.048811:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016342.048814:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016342.048816:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016342.048819:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 608 00000100:00080000:0.0:1734016342.048822:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016342.048826:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:3.0:1734016342.048827:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334083584:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734016342.048830:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016342.048840:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334083968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016342.048849:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016342.048852:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016342.048854:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016342.048857:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 608 00000100:00080000:0.0:1734016342.048860:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016342.048863:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016342.048868:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016342.048876:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334084096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016342.048883:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016342.048886:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016342.048888:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016342.048891:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 608 00000800:00000100:3.0:1734016342.048891:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734016342.048894:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016342.048898:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016342.048901:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016342.048909:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334084224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016342.048911:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016342/real 1734016342] req@ffff8800a934e300 x1818247334083584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016342.048916:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016342.048920:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016342.048922:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016342.048925:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 608 00000100:00080000:0.0:1734016342.048928:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016342.048931:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016342.048935:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016342.048938:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334083584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016342.048943:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334084352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016342.048948:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016342.048951:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016342.048956:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334083584:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016342.048963:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334083712:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016342.048978:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016342.048985:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016342/real 1734016342] req@ffff8800a934e680 x1818247334083712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016342.048996:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334083712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016342.049003:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016342.049006:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016342.049009:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334083712:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016342.049015:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334083840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016342.049028:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016342.049034:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016342/real 1734016342] req@ffff8800a934f100 x1818247334083840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016342.049044:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334083840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016342.049052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016342.049055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016342.049058:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334083840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016342.049081:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334083968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016342.049094:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016342.049101:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016342/real 1734016342] req@ffff8800a934fb80 x1818247334083968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016342.049111:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334083968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016342.049118:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016342.049121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016342.049124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334083968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016342.049129:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334084096:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016342.049142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016342.049147:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016342/real 1734016342] req@ffff8800a934d500 x1818247334084096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016342.049157:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334084096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016342.049164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016342.049166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016342.049169:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334084096:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016342.049175:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334084224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016342.049190:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016342.049196:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016342/real 1734016342] req@ffff8800a934df80 x1818247334084224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016342.049207:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334084224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016342.049214:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016342.049216:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016342.049219:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334084224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016342.049225:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334084352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016342.049236:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016342.049242:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016342/real 1734016342] req@ffff8800a934ce00 x1818247334084352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016342.049252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334084352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016342.049259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016342.049261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016342.049264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334084352:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734016347.056607:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016347.056616:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016347.056621:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016347.056632:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 613 00000100:00080000:0.0:1734016347.056640:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016347.056648:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016347.056710:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334084480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016347.056743:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016347.056747:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016347.056749:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016347.056754:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 613 00000100:00080000:0.0:1734016347.056757:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016347.056761:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016347.056771:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016347.056782:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334084608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016347.056793:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016347.056803:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016347.056805:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016347.056809:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 613 00000100:00080000:0.0:1734016347.056812:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016347.056816:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016347.056821:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016347.056830:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f480 x1818247334084736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016347.056839:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016347.056842:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016347.056844:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016347.056847:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 613 00000100:00080000:0.0:1734016347.056850:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016347.056853:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016347.056857:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734016347.056863:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334084480:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016347.056865:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334084864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016347.056873:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016347.056876:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016347.056878:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016347.056881:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 613 00000100:00080000:0.0:1734016347.056884:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016347.056887:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016347.056891:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016347.056898:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334084992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016347.056906:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016347.056909:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016347.056910:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016347.056913:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 613 00000100:00080000:0.0:1734016347.056916:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016347.056919:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016347.056923:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1734016347.056924:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016347.056933:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334085120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016347.056941:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016347.056944:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1734016347.056944:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016347/real 1734016347] req@ffff8800a934ea00 x1818247334084480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016347.056945:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016347.056949:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 613 00000100:00080000:0.0:1734016347.056951:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016347.056954:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016347.056958:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016347.056966:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334085248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:3.0:1734016347.056971:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334084480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016347.056981:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016347.056984:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016347.056989:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334084480:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016347.056996:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334084608:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016347.057012:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016347.057019:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016347/real 1734016347] req@ffff8800a934c380 x1818247334084608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016347.057030:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334084608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016347.057038:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016347.057040:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016347.057043:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334084608:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016347.057049:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334084736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016347.057062:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016347.057068:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016347/real 1734016347] req@ffff8800a934f480 x1818247334084736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016347.057078:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f480 x1818247334084736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016347.057085:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016347.057088:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016347.057091:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334084736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016347.057115:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334084864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016347.057128:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016347.057134:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016347/real 1734016347] req@ffff8800a934ca80 x1818247334084864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016347.057145:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334084864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016347.057152:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016347.057154:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016347.057157:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334084864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016347.057162:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334084992:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016347.057174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016347.057180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016347/real 1734016347] req@ffff8800a934ed80 x1818247334084992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016347.057190:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334084992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016347.057197:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016347.057199:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016347.057202:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334084992:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016347.057207:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334085120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016347.057223:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016347.057229:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016347/real 1734016347] req@ffff8800a934c700 x1818247334085120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016347.057239:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334085120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016347.057245:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016347.057248:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016347.057250:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334085120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016347.057256:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334085248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016347.057267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016347.057273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016347/real 1734016347] req@ffff8800a934d880 x1818247334085248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016347.057283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334085248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016347.057289:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016347.057292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016347.057294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334085248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016352.064897:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334085376:192.168.201.116@tcp:250:kworker.0 00000800:00000100:3.0:1734016352.064959:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016352.064980:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016352/real 1734016352] req@ffff8800a78a1500 x1818247334085376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016352.065007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334085376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016352.065017:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016352.065020:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016352.065025:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334085376:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016352.065032:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334085504:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016352.065048:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016352.065055:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016352/real 1734016352] req@ffff8800a78a1880 x1818247334085504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016352.065066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334085504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016352.065074:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016352.065076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016352.065080:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334085504:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016352.065085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334085632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016352.065098:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016352.065105:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016352/real 1734016352] req@ffff8800a78a1f80 x1818247334085632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016352.065115:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334085632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016352.065122:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016352.065125:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016352.065128:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334085632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016352.065153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334085760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016352.065166:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016352.065172:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016352/real 1734016352] req@ffff8800a78a2300 x1818247334085760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016352.065182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334085760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016352.065190:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016352.065192:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016352.065195:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334085760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016352.065200:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334085888:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016352.065212:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016352.065218:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016352/real 1734016352] req@ffff8800a78a3480 x1818247334085888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016352.065227:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334085888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016352.065234:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016352.065236:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016352.065239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334085888:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016352.065245:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334086016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016352.065260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016352.065266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016352/real 1734016352] req@ffff8800a78a2680 x1818247334086016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016352.065276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334086016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016352.065283:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016352.065285:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016352.065288:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334086016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016352.065293:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334086144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016352.065305:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016352.065311:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016352/real 1734016352] req@ffff8800a78a0700 x1818247334086144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016352.065321:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334086144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016352.065327:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016352.065329:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016352.065332:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334086144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016357.072824:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334086272:192.168.201.116@tcp:250:kworker.0 00000800:00000100:3.0:1734016357.072889:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016357.072910:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016357/real 1734016357] req@ffff8800a78a0a80 x1818247334086272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016357.072937:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334086272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016357.072947:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016357.072950:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016357.072955:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334086272:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016357.072962:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334086400:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016357.072977:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016357.072984:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016357/real 1734016357] req@ffff8800a78a2d80 x1818247334086400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016357.072995:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334086400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016357.073003:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016357.073005:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016357.073009:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334086400:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016357.073014:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334086528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016357.073028:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016357.073034:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016357/real 1734016357] req@ffff8800a78a3b80 x1818247334086528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016357.073044:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334086528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016357.073051:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016357.073054:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016357.073058:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334086528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016357.073083:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334086656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016357.073096:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016357.073102:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016357/real 1734016357] req@ffff8800a78a0700 x1818247334086656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016357.073113:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334086656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016357.073120:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016357.073122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016357.073125:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334086656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016357.073130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334086784:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016357.073142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016357.073148:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016357/real 1734016357] req@ffff8800a78a2680 x1818247334086784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016357.073158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334086784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016357.073165:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016357.073167:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016357.073170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334086784:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016357.073176:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334086912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016357.073191:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016357.073197:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016357/real 1734016357] req@ffff8800a78a3480 x1818247334086912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016357.073207:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334086912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016357.073214:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016357.073216:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016357.073219:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334086912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016357.073224:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334087040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016357.073236:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016357.073242:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016357/real 1734016357] req@ffff8800a78a2300 x1818247334087040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016357.073252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334087040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016357.073259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016357.073261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016357.073264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334087040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016362.080892:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334087168:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016362.080994:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334087296:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016362.081016:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334087424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016362.081040:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334087552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016362.081056:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334087680:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016362.081073:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334087808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016362.081089:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334087936:192.168.201.116@tcp:8:kworker.0 00000400:00000100:0.0:1734016365.086802:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:0.0:1734016365.086861:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734016365.086886:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016365.086930:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016365.086939:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016365.086948:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016365.086956:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016365.086963:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016365.086970:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:3.0:1734016365.087044:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016362/real 1734016365] req@ffff8800a78a2300 x1818247334087552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016365.087077:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334087552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016365.087089:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016365.087093:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016365.087098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334087552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016365.087107:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016362/real 1734016365] req@ffff8800a78a3480 x1818247334087680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016365.087119:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334087680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016365.087127:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016365.087130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016365.087133:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334087680:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016365.087140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016362/real 1734016365] req@ffff8800a78a2680 x1818247334087808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016365.087151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334087808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016365.087160:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016365.087162:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016365.087166:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334087808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016365.087172:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016362/real 1734016365] req@ffff8800a78a0700 x1818247334087936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016365.087183:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334087936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016365.087190:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016365.087192:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016365.087195:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334087936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016365.087202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016362/real 1734016365] req@ffff8800a78a1f80 x1818247334087168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016365.087213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334087168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016365.087220:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016365.087223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016365.087226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334087168:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016365.087233:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016362/real 1734016365] req@ffff8800a78a1880 x1818247334087296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016365.087243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334087296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016365.087258:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016365.087261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016365.087264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334087296:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016365.087271:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016362/real 1734016365] req@ffff8800a78a1500 x1818247334087424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016365.087281:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334087424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016365.087288:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016365.087291:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016365.087294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334087424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016367.088888:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334088064:192.168.201.116@tcp:250:kworker.0 00000800:00000100:3.0:1734016367.088964:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016367.088986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016367/real 1734016367] req@ffff8800a78a3b80 x1818247334088064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016367.089006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334088064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016367.089017:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016367.089021:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016367.089027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334088064:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016367.089035:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334088192:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016367.089053:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016367.089061:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016367/real 1734016367] req@ffff8800a78a2d80 x1818247334088192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016367.089072:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334088192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016367.089080:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016367.089082:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016367.089086:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334088192:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016367.089092:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334088320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016367.089106:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016367.089112:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016367/real 1734016367] req@ffff8800a78a0a80 x1818247334088320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016367.089122:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334088320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016367.089129:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016367.089132:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016367.089135:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334088320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016367.089162:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334088448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016367.089175:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016367.089182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016367/real 1734016367] req@ffff8800a78a1500 x1818247334088448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016367.089192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334088448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016367.089207:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016367.089210:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016367.089213:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334088448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016367.089218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334088576:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016367.089231:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016367.089237:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016367/real 1734016367] req@ffff8800a78a1880 x1818247334088576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016367.089247:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334088576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016367.089254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016367.089256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016367.089260:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334088576:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016367.089266:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334088704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016367.089278:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016367.089284:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016367/real 1734016367] req@ffff8800a78a1f80 x1818247334088704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016367.089293:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334088704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016367.089300:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016367.089303:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016367.089306:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334088704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016367.089312:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334088832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016367.089372:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016367.089381:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016367/real 1734016367] req@ffff8800a78a0700 x1818247334088832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016367.089392:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334088832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016367.089400:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016367.089402:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016367.089405:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334088832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016372.096803:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334088960:192.168.201.116@tcp:250:kworker.0 00000800:00000100:3.0:1734016372.096871:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016372.096893:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016372/real 1734016372] req@ffff8800a78a2680 x1818247334088960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016372.096912:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334088960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016372.096922:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016372.096925:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016372.096930:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334088960:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734016372.096937:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334089088:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016372.096954:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016372.096960:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016372/real 1734016372] req@ffff8800a78a3480 x1818247334089088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016372.096979:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334089088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016372.096987:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016372.096989:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016372.096993:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334089088:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016372.096999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334089216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016372.097012:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016372.097018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016372/real 1734016372] req@ffff8800a78a2300 x1818247334089216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016372.097028:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334089216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016372.097036:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016372.097038:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016372.097041:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334089216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016372.097065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334089344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016372.097079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016372.097085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016372/real 1734016372] req@ffff8800a78a0700 x1818247334089344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016372.097096:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334089344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016372.097103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016372.097105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016372.097109:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334089344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016372.097114:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334089472:192.168.201.116@tcp:38:kworker.0 00000800:00000100:3.0:1734016372.097127:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016372.097133:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016372/real 1734016372] req@ffff8800a78a1f80 x1818247334089472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016372.097142:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334089472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016372.097149:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016372.097152:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016372.097154:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334089472:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734016372.097160:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334089600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016372.097172:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016372.097177:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016372/real 1734016372] req@ffff8800a78a1880 x1818247334089600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016372.097187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334089600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016372.097194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016372.097196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016372.097199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334089600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734016372.097204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334089728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:3.0:1734016372.097220:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734016372.097226:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016372/real 1734016372] req@ffff8800a78a1500 x1818247334089728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016372.097237:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334089728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016372.097276:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016372.097278:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734016372.097282:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334089728:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016377.104760:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016377.104771:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334089856:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016377.104838:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016377.104864:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016377/real 1734016377] req@ffff8800a78a0a80 x1818247334089856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016377.104885:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334089856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016377.104895:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016377.104899:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016377.104904:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334089856:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016377.104928:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334089984:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016377.104945:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016377.104952:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016377/real 1734016377] req@ffff8800a78a2d80 x1818247334089984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016377.104964:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334089984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016377.104973:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016377.104976:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016377.104979:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334089984:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016377.104986:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334090112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016377.104999:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016377.105005:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016377/real 1734016377] req@ffff8800a78a3b80 x1818247334090112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016377.105015:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334090112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016377.105022:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016377.105025:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016377.105028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334090112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016377.105034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334090240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016377.105047:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016377.105053:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016377/real 1734016377] req@ffff8800a78a1500 x1818247334090240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016377.105063:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334090240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016377.105070:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016377.105072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016377.105075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334090240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016377.105086:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334090368:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016377.105099:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016377.105105:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016377/real 1734016377] req@ffff8800a78a1880 x1818247334090368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016377.105116:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334090368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016377.105123:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016377.105125:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016377.105128:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334090368:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016377.105133:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334090496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016377.105145:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016377.105151:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016377/real 1734016377] req@ffff8800a78a1f80 x1818247334090496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016377.105208:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334090496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016377.105215:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016377.105217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016377.105220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334090496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016377.105244:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334090624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016377.105262:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016377.105269:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016377/real 1734016377] req@ffff8800a78a0700 x1818247334090624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016377.105281:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334090624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016377.105288:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016377.105290:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016377.105293:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334090624:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016380.110784:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016382.112826:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334090752:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016382.112894:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016382.112920:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016382/real 1734016382] req@ffff8800a78a2300 x1818247334090752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016382.112940:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334090752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016382.112951:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016382.112955:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016382.112959:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334090752:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016382.112967:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334090880:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016382.112984:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016382.112991:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016382/real 1734016382] req@ffff8800a78a3480 x1818247334090880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016382.113002:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334090880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016382.113010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016382.113013:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016382.113016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334090880:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016382.113039:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334091008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016382.113053:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016382.113060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016382/real 1734016382] req@ffff8800a78a2680 x1818247334091008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016382.113105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334091008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016382.113113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016382.113116:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016382.113119:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334091008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016382.113125:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334091136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016382.113141:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016382.113148:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016382/real 1734016382] req@ffff8800a78a0700 x1818247334091136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016382.113159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334091136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016382.113167:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016382.113169:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016382.113172:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334091136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016382.113177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334091264:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016382.113193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016382.113200:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016382/real 1734016382] req@ffff8800a78a1f80 x1818247334091264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016382.113211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334091264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016382.113218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016382.113220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016382.113223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334091264:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016382.113228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334091392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016382.113240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016382.113247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016382/real 1734016382] req@ffff8800a78a1880 x1818247334091392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016382.113257:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334091392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016382.113265:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016382.113267:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016382.113270:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334091392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016382.113275:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334091520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016382.113288:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016382.113295:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016382/real 1734016382] req@ffff8800a78a1500 x1818247334091520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016382.113311:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334091520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016382.113318:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016382.113321:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016382.113323:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334091520:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016383.116657:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016386.122773:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016387.120817:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334091648:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016387.120883:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016387.120910:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016387/real 1734016387] req@ffff8800a78a3b80 x1818247334091648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016387.120930:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334091648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016387.120941:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016387.120945:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016387.120949:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334091648:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016387.120957:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334091776:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016387.120973:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016387.120981:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016387/real 1734016387] req@ffff8800a78a2d80 x1818247334091776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016387.121025:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334091776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016387.121033:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016387.121037:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016387.121040:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334091776:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016387.121047:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334091904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016387.121064:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016387.121071:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016387/real 1734016387] req@ffff8800a78a0a80 x1818247334091904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016387.121082:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334091904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016387.121089:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016387.121092:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016387.121095:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334091904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016387.121118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334092032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016387.121130:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016387.121137:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016387/real 1734016387] req@ffff8800a78a1500 x1818247334092032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016387.121147:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334092032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016387.121155:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016387.121157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016387.121160:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334092032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016387.121166:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334092160:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016387.121181:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016387.121187:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016387/real 1734016387] req@ffff8800a78a1880 x1818247334092160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016387.121198:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334092160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016387.121205:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016387.121207:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016387.121210:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334092160:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016387.121215:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334092288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016387.121227:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016387.121233:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016387/real 1734016387] req@ffff8800a78a1f80 x1818247334092288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016387.121243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334092288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016387.121250:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016387.121252:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016387.121255:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334092288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016387.121260:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334092416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016387.121272:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016387.121278:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016387/real 1734016387] req@ffff8800a78a0700 x1818247334092416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016387.121295:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334092416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016387.121302:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016387.121304:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016387.121307:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334092416:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016389.128749:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016392.128842:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334092544:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016392.128937:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016392.128965:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016392/real 1734016392] req@ffff8800a78a2680 x1818247334092544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016392.128985:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334092544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016392.128996:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016392.128999:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016392.129005:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334092544:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016392.129012:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334092672:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016392.129031:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016392.129038:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016392/real 1734016392] req@ffff8800a78a3480 x1818247334092672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016392.129049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334092672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016392.129057:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016392.129061:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016392.129064:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334092672:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016392.129071:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334092800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016392.129084:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016392.129091:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016392/real 1734016392] req@ffff8800a78a2300 x1818247334092800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016392.129101:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334092800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016392.129109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016392.129111:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016392.129115:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334092800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016392.129143:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334092928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016392.129155:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016392.129162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016392/real 1734016392] req@ffff8800a78a0700 x1818247334092928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016392.129172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334092928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016392.129180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016392.129182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016392.129185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334092928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016392.129191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334093056:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016392.129206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016392.129213:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016392/real 1734016392] req@ffff8800a78a1f80 x1818247334093056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016392.129223:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334093056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016392.129230:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016392.129232:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016392.129235:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334093056:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016392.129240:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334093184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016392.129251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016392.129258:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016392/real 1734016392] req@ffff8800a78a1880 x1818247334093184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016392.129268:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334093184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016392.129275:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016392.129278:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016392.129281:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334093184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016392.129286:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334093312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016392.129299:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016392.129305:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016392/real 1734016392] req@ffff8800a78a1500 x1818247334093312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016392.129321:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334093312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016392.129328:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016392.129330:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016392.129333:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334093312:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016392.134759:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016395.140776:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016397.136845:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334093440:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016397.136912:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016397.136939:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016397/real 1734016397] req@ffff8800a78a0a80 x1818247334093440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016397.136960:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334093440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016397.136971:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016397.136974:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016397.136978:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334093440:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016397.136986:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334093568:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016397.137003:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016397.137010:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016397/real 1734016397] req@ffff8800a78a2d80 x1818247334093568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016397.137022:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334093568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016397.137029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016397.137033:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016397.137036:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334093568:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016397.137042:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334093696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016397.137056:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016397.137063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016397/real 1734016397] req@ffff8800a78a3b80 x1818247334093696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016397.137073:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334093696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016397.137081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016397.137083:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016397.137086:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334093696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016397.137111:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334093824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016397.137124:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016397.137130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016397/real 1734016397] req@ffff8800a78a1500 x1818247334093824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016397.137141:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334093824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016397.137149:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016397.137151:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016397.137154:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334093824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016397.137159:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334093952:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016397.137174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016397.137181:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016397/real 1734016397] req@ffff8800a78a1880 x1818247334093952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016397.137191:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334093952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016397.137198:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016397.137200:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016397.137203:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334093952:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016397.137208:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334094080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016397.137220:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016397.137227:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016397/real 1734016397] req@ffff8800a78a1f80 x1818247334094080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016397.137236:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334094080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016397.137243:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016397.137245:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016397.137248:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334094080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016397.137253:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334094208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016397.137266:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016397.137272:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016397/real 1734016397] req@ffff8800a78a0700 x1818247334094208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016397.137288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334094208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016397.137296:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016397.137298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016397.137301:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334094208:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016398.146755:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016401.152814:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016402.144823:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334094336:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016402.144889:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016402.144916:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016402/real 1734016402] req@ffff8800a78a2300 x1818247334094336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016402.144936:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334094336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016402.144947:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016402.144950:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016402.144955:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334094336:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016402.144962:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334094464:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016402.144979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016402.144986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016402/real 1734016402] req@ffff8800a78a3480 x1818247334094464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016402.144997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334094464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016402.145005:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016402.145009:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016402.145012:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334094464:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016402.145034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334094592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016402.145049:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016402.145055:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016402/real 1734016402] req@ffff8800a78a2680 x1818247334094592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016402.145066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334094592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016402.145073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016402.145076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016402.145079:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334094592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016402.145085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334094720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016402.145097:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016402.145103:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016402/real 1734016402] req@ffff8800a78a0700 x1818247334094720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016402.145113:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334094720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016402.145120:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016402.145122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016402.145125:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334094720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016402.145130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334094848:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016402.145146:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016402.145153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016402/real 1734016402] req@ffff8800a78a1f80 x1818247334094848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016402.145163:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334094848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016402.145170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016402.145172:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016402.145175:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334094848:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016402.145180:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334094976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016402.145193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016402.145199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016402/real 1734016402] req@ffff8800a78a1880 x1818247334094976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016402.145209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334094976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016402.145215:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016402.145218:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016402.145220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334094976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016402.145226:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334095104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016402.145238:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016402.145245:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016402/real 1734016402] req@ffff8800a78a1500 x1818247334095104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016402.145261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334095104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016402.145268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016402.145270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016402.145273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334095104:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016404.158774:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016407.152852:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334095232:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016407.152916:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016407.152943:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016407/real 1734016407] req@ffff8800a78a3b80 x1818247334095232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016407.152963:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334095232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016407.152974:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016407.152977:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016407.152982:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334095232:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016407.152989:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334095360:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016407.153007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016407.153014:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016407/real 1734016407] req@ffff8800a78a2d80 x1818247334095360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016407.153025:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334095360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016407.153032:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016407.153036:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016407.153039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334095360:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016407.153045:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334095488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016407.153059:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016407.153065:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016407/real 1734016407] req@ffff8800a78a0a80 x1818247334095488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016407.153075:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334095488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016407.153083:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016407.153085:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016407.153089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334095488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016407.153112:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334095616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016407.153125:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016407.153132:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016407/real 1734016407] req@ffff8800a78a1500 x1818247334095616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016407.153142:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334095616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016407.153150:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016407.153152:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016407.153155:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334095616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016407.153160:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334095744:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016407.153176:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016407.153182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016407/real 1734016407] req@ffff8800a78a1880 x1818247334095744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016407.153192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334095744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016407.153199:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016407.153202:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016407.153205:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334095744:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016407.153210:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334095872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016407.153222:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016407.153228:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016407/real 1734016407] req@ffff8800a78a1f80 x1818247334095872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016407.153238:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334095872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016407.153245:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016407.153247:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016407.153250:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334095872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016407.153255:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334096000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016407.153267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016407.153273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016407/real 1734016407] req@ffff8800a78a0700 x1818247334096000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016407.153290:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334096000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016407.153296:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016407.153298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016407.153301:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334096000:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016407.164733:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016410.170842:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016412.160878:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334096128:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016412.160944:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016412.160970:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016412/real 1734016412] req@ffff8800a78a2680 x1818247334096128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016412.160990:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334096128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016412.161001:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016412.161004:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016412.161009:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334096128:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016412.161016:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334096256:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016412.161032:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016412.161039:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016412/real 1734016412] req@ffff8800a78a3480 x1818247334096256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016412.161051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334096256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016412.161058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016412.161062:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016412.161065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334096256:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016412.161071:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334096384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016412.161084:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016412.161091:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016412/real 1734016412] req@ffff8800a78a2300 x1818247334096384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016412.161101:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334096384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016412.161109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016412.161111:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016412.161115:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334096384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016412.161138:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334096512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016412.161151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016412.161158:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016412/real 1734016412] req@ffff8800a78a0700 x1818247334096512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016412.161168:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334096512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016412.161176:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016412.161178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016412.161181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334096512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016412.161186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334096640:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016412.161201:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016412.161208:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016412/real 1734016412] req@ffff8800a78a1f80 x1818247334096640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016412.161218:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334096640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016412.161225:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016412.161227:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016412.161230:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334096640:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016412.161236:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334096768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016412.161247:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016412.161253:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016412/real 1734016412] req@ffff8800a78a1880 x1818247334096768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016412.161263:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334096768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016412.161270:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016412.161272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016412.161275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334096768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016412.161280:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334096896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016412.161292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016412.161298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016412/real 1734016412] req@ffff8800a78a1500 x1818247334096896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016412.161315:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334096896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016412.161322:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016412.161325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016412.161327:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334096896:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016413.176748:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016416.182774:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016417.168824:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334097024:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016417.168889:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016417.168915:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016417/real 1734016417] req@ffff8800a78a0a80 x1818247334097024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016417.168936:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334097024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016417.168946:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016417.168950:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016417.168954:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334097024:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016417.168961:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334097152:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016417.168978:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016417.168986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016417/real 1734016417] req@ffff8800a78a2d80 x1818247334097152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016417.168997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334097152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016417.169004:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016417.169008:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016417.169012:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334097152:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016417.169018:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334097280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016417.169032:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016417.169038:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016417/real 1734016417] req@ffff8800a78a3b80 x1818247334097280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016417.169048:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334097280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016417.169056:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016417.169059:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016417.169062:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334097280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016417.169086:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334097408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016417.169099:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016417.169105:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016417/real 1734016417] req@ffff8800a78a1500 x1818247334097408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016417.169116:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334097408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016417.169124:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016417.169126:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016417.169129:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334097408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016417.169134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334097536:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016417.169149:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016417.169156:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016417/real 1734016417] req@ffff8800a78a1880 x1818247334097536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016417.169166:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334097536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016417.169173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016417.169175:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016417.169178:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334097536:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016417.169183:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334097664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016417.169195:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016417.169201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016417/real 1734016417] req@ffff8800a78a1f80 x1818247334097664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016417.169211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334097664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016417.169217:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016417.169220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016417.169223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334097664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016417.169228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334097792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016417.169241:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016417.169247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016417/real 1734016417] req@ffff8800a78a0700 x1818247334097792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016417.169263:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334097792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016417.169270:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016417.169272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016417.169275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334097792:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016419.188742:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016422.176831:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334097920:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016422.176895:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016422.176922:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016422/real 1734016422] req@ffff8800a78a2300 x1818247334097920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016422.176942:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334097920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016422.176953:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016422.176956:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016422.176961:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334097920:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016422.176968:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334098048:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016422.176985:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016422.176992:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016422/real 1734016422] req@ffff8800a78a3480 x1818247334098048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016422.177004:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334098048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016422.177011:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016422.177015:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016422.177018:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334098048:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016422.177025:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334098176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016422.177038:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016422.177044:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016422/real 1734016422] req@ffff8800a78a2680 x1818247334098176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016422.177054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334098176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016422.177062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016422.177065:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016422.177068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334098176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016422.177090:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334098304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016422.177103:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016422.177109:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016422/real 1734016422] req@ffff8800a78a0700 x1818247334098304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016422.177120:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334098304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016422.177127:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016422.177130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016422.177133:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334098304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016422.177138:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334098432:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016422.177153:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016422.177160:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016422/real 1734016422] req@ffff8800a78a1f80 x1818247334098432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016422.177170:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334098432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016422.177178:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016422.177180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016422.177182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334098432:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016422.177188:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334098560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016422.177200:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016422.177206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016422/real 1734016422] req@ffff8800a78a1880 x1818247334098560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016422.177215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334098560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016422.177222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016422.177224:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016422.177227:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334098560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016422.177233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334098688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016422.177245:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016422.177251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016422/real 1734016422] req@ffff8800a78a1500 x1818247334098688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016422.177268:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334098688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016422.177275:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016422.177277:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016422.177280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334098688:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016422.194717:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016425.200780:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016427.184833:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334098816:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016427.184928:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334098944:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016427.184949:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334099072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016427.184972:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334099200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016427.184987:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334099328:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016427.185001:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334099456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016427.185015:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334099584:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016428.206730:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734016430.190710:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734016430.190756:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734016430.190775:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016430.190813:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016430.190821:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016430.190827:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016430.190834:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016430.190841:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016430.190847:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:0.0:1734016430.190917:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016427/real 1734016430] req@ffff8800a78a1500 x1818247334099200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016430.190958:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334099200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016430.190975:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016430.190981:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016430.190988:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334099200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016430.191001:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016427/real 1734016430] req@ffff8800a78a1880 x1818247334099328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016430.191020:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334099328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016430.191033:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016430.191036:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016430.191042:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334099328:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016430.191052:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016427/real 1734016430] req@ffff8800a78a1f80 x1818247334099456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016430.191070:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334099456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016430.191083:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016430.191087:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016430.191092:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334099456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016430.191103:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016427/real 1734016430] req@ffff8800a78a0700 x1818247334099584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016430.191121:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334099584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016430.191134:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016430.191138:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016430.191143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334099584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016430.191157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016427/real 1734016430] req@ffff8800a78a3b80 x1818247334098816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016430.191177:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334098816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016430.191188:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016430.191192:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016430.191197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334098816:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016430.191207:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016427/real 1734016430] req@ffff8800a78a2d80 x1818247334098944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016430.191225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334098944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016430.191237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016430.191241:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016430.191247:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334098944:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016430.191318:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016427/real 1734016430] req@ffff8800a78a0a80 x1818247334099072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016430.191339:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334099072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016430.191352:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016430.191355:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016430.191361:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334099072:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016431.212781:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016432.192782:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334099712:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016432.192846:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016432.192869:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016432/real 1734016432] req@ffff8800a78a2680 x1818247334099712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016432.192888:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334099712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016432.192898:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016432.192902:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016432.192907:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334099712:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016432.192914:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334099840:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016432.192931:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016432.192939:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016432/real 1734016432] req@ffff8800a78a3480 x1818247334099840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016432.192951:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334099840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016432.192958:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016432.192961:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016432.192964:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334099840:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016432.192986:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334099968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016432.193000:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016432.193006:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016432/real 1734016432] req@ffff8800a78a2300 x1818247334099968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016432.193016:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334099968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016432.193024:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016432.193027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016432.193030:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334099968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016432.193036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334100096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016432.193048:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016432.193054:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016432/real 1734016432] req@ffff8800a78a0a80 x1818247334100096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016432.193064:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334100096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016432.193071:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016432.193073:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016432.193077:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334100096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016432.193083:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334100224:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016432.193095:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016432.193101:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016432/real 1734016432] req@ffff8800a78a2d80 x1818247334100224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016432.193111:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334100224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016432.193119:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016432.193121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016432.193124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334100224:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016432.193131:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334100352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016432.193143:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016432.193150:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016432/real 1734016432] req@ffff8800a78a3b80 x1818247334100352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016432.193160:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334100352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016432.193166:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016432.193169:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016432.193172:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334100352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016432.193177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334100480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016432.193189:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016432.193195:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016432/real 1734016432] req@ffff8800a78a0700 x1818247334100480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016432.193205:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334100480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016432.193212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016432.193255:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016432.193260:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334100480:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016434.218692:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016437.200837:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334100608:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016437.200904:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016437.200929:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016437/real 1734016437] req@ffff8800a78a1f80 x1818247334100608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016437.200950:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334100608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016437.200961:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016437.200964:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016437.200969:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334100608:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016437.200977:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334100736:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016437.200993:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016437.201000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016437/real 1734016437] req@ffff8800a78a1880 x1818247334100736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016437.201012:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334100736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016437.201019:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016437.201022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016437.201025:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334100736:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016437.201031:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334100864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016437.201044:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016437.201050:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016437/real 1734016437] req@ffff8800a78a1500 x1818247334100864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016437.201060:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334100864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016437.201067:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016437.201070:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016437.201073:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334100864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016437.201097:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334100992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016437.201110:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016437.201117:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016437/real 1734016437] req@ffff8800a78a0700 x1818247334100992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016437.201127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334100992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016437.201168:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016437.201171:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016437.201175:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334100992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016437.201181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334101120:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016437.201198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016437.201206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016437/real 1734016437] req@ffff8800a78a3b80 x1818247334101120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016437.201217:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334101120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016437.201224:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016437.201227:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016437.201230:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334101120:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016437.201235:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334101248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016437.201250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016437.201256:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016437/real 1734016437] req@ffff8800a78a2d80 x1818247334101248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016437.201267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334101248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016437.201274:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016437.201276:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016437.201279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334101248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016437.201285:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334101376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016437.201297:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016437.201303:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016437/real 1734016437] req@ffff8800a78a0a80 x1818247334101376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016437.201312:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334101376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016437.201319:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016437.201321:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016437.201324:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334101376:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016437.224761:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016440.230805:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016442.208785:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334101504:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016442.208851:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016442.208877:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016442/real 1734016442] req@ffff8800a78a2300 x1818247334101504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016442.208905:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334101504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016442.208916:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016442.208919:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016442.208924:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334101504:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016442.208931:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334101632:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016442.208949:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016442.208956:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016442/real 1734016442] req@ffff8800a78a3480 x1818247334101632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016442.208967:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334101632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016442.208975:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016442.208978:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016442.208981:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334101632:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016442.209004:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334101760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016442.209017:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016442.209024:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016442/real 1734016442] req@ffff8800a78a2680 x1818247334101760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016442.209034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334101760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016442.209042:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016442.209105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016442.209110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334101760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016442.209118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334101888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016442.209139:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016442.209147:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016442/real 1734016442] req@ffff8800a78a0a80 x1818247334101888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016442.209159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334101888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016442.209166:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016442.209168:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016442.209171:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334101888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016442.209177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334102016:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016442.209189:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016442.209195:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016442/real 1734016442] req@ffff8800a78a2d80 x1818247334102016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016442.209205:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334102016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016442.209213:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016442.209215:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016442.209218:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334102016:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016442.209224:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334102144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016442.209239:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016442.209245:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016442/real 1734016442] req@ffff8800a78a3b80 x1818247334102144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016442.209256:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334102144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016442.209263:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016442.209265:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016442.209268:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334102144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016442.209273:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334102272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016442.209285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016442.209292:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016442/real 1734016442] req@ffff8800a78a0700 x1818247334102272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016442.209301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334102272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016442.209308:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016442.209310:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016442.209313:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334102272:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016443.236712:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016446.242746:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016447.216838:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334102400:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016447.216904:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016447.216929:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016447/real 1734016447] req@ffff8800a78a1500 x1818247334102400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016447.216957:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334102400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016447.216998:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016447.217001:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016447.217006:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334102400:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016447.217014:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334102528:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016447.217035:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016447.217043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016447/real 1734016447] req@ffff8800a78a1880 x1818247334102528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016447.217054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334102528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016447.217061:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016447.217064:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016447.217067:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334102528:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016447.217073:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334102656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016447.217087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016447.217093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016447/real 1734016447] req@ffff8800a78a1f80 x1818247334102656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016447.217104:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334102656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016447.217111:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016447.217114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016447.217117:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334102656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016447.217141:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334102784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016447.217154:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016447.217160:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016447/real 1734016447] req@ffff8800a78a0700 x1818247334102784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016447.217171:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334102784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016447.217178:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016447.217180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016447.217183:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334102784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016447.217188:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334102912:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016447.217200:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016447.217206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016447/real 1734016447] req@ffff8800a78a3b80 x1818247334102912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016447.217216:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334102912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016447.217223:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016447.217225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016447.217228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334102912:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016447.217234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334103040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016447.217250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016447.217256:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016447/real 1734016447] req@ffff8800a78a2d80 x1818247334103040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016447.217266:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334103040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016447.217273:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016447.217276:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016447.217279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334103040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016447.217284:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334103168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016447.217296:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016447.217302:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016447/real 1734016447] req@ffff8800a78a0a80 x1818247334103168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016447.217312:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334103168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016447.217319:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016447.217321:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016447.217324:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334103168:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016449.248762:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016452.224733:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334103296:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016452.224809:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016452.224834:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016452/real 1734016452] req@ffff8800a78a2680 x1818247334103296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016452.224868:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334103296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016452.224903:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016452.224906:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016452.224929:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334103296:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016452.224936:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334103424:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016452.224965:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016452.224973:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016452/real 1734016452] req@ffff8800a78a3480 x1818247334103424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016452.224984:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334103424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016452.224990:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016452.224993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016452.225002:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334103424:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016452.225023:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334103552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016452.225037:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016452.225050:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016452/real 1734016452] req@ffff8800a78a2300 x1818247334103552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016452.225060:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334103552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016452.225066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016452.225069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016452.225072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334103552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016452.225077:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334103680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016452.225095:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016452.225101:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016452/real 1734016452] req@ffff8800a78a0a80 x1818247334103680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016452.225110:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334103680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016452.225117:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016452.225119:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016452.225121:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334103680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016452.225126:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334103808:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016452.225144:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016452.225149:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016452/real 1734016452] req@ffff8800a78a2d80 x1818247334103808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016452.225158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334103808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016452.225164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016452.225166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016452.225174:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334103808:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016452.225180:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334103936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016452.225194:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016452.225206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016452/real 1734016452] req@ffff8800a78a3b80 x1818247334103936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016452.225216:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334103936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016452.225222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016452.225224:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016452.225227:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334103936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016452.225232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334104064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016452.225243:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016452.225254:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016452/real 1734016452] req@ffff8800a78a0700 x1818247334104064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016452.225263:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334104064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016452.225269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016452.225271:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016452.225273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334104064:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016452.254656:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016455.260756:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016457.232699:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334104192:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016457.232742:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016457.232759:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016457/real 1734016457] req@ffff8800a78a1f80 x1818247334104192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016457.232774:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334104192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016457.232780:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016457.232782:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016457.232784:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334104192:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016457.232807:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334104320:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016457.232819:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016457.232823:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016457/real 1734016457] req@ffff8800a78a1880 x1818247334104320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016457.232829:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334104320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016457.232833:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016457.232835:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016457.232836:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334104320:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016457.232851:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334104448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016457.232859:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016457.232862:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016457/real 1734016457] req@ffff8800a78a1500 x1818247334104448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016457.232867:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334104448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016457.232871:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016457.232873:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016457.232874:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334104448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016457.232877:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334104576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016457.232884:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016457.232887:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016457/real 1734016457] req@ffff8800a78a0700 x1818247334104576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016457.232892:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334104576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016457.232896:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016457.232897:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016457.232899:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334104576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016457.232901:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334104704:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016457.232908:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016457.232911:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016457/real 1734016457] req@ffff8800a78a3b80 x1818247334104704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016457.232916:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334104704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016457.232920:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016457.232921:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016457.232923:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334104704:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016457.232926:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334104832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016457.232934:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016457.232937:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016457/real 1734016457] req@ffff8800a78a2d80 x1818247334104832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016457.232942:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334104832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016457.232946:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016457.232947:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016457.232948:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334104832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016457.232951:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334104960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016457.232957:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016457.232960:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016457/real 1734016457] req@ffff8800a78a0a80 x1818247334104960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016457.232965:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334104960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016457.232968:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016457.232969:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016457.232971:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334104960:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016458.266725:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016461.272666:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016462.240661:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334105088:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016462.240780:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016462.240802:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016462/real 1734016462] req@ffff8800a78a2300 x1818247334105088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016462.240824:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334105088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016462.240849:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016462.240853:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016462.240857:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334105088:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016462.240865:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334105216:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016462.240905:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016462.240916:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016462/real 1734016462] req@ffff8800a78a3480 x1818247334105216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016462.240926:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334105216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016462.240933:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016462.240935:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016462.240939:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334105216:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016462.240962:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334105344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016462.240979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016462.241007:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016462/real 1734016462] req@ffff8800a78a2680 x1818247334105344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016462.241018:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334105344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016462.241024:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016462.241026:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016462.241030:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334105344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016462.241036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334105472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016462.241052:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016462.241059:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016462/real 1734016462] req@ffff8800a78a0a80 x1818247334105472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016462.241069:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334105472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016462.241075:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016462.241077:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016462.241080:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334105472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016462.241084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334105600:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016462.241098:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016462.241105:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016462/real 1734016462] req@ffff8800a78a2d80 x1818247334105600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016462.241113:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334105600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016462.241119:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016462.241121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016462.241124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334105600:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016462.241144:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334105728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016462.241162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016462.241184:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016462/real 1734016462] req@ffff8800a78a3b80 x1818247334105728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016462.241203:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334105728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016462.241211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016462.241212:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016462.241216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334105728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016462.241221:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334105856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016462.241254:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016462.241262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016462/real 1734016462] req@ffff8800a78a0700 x1818247334105856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016462.241271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334105856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016462.241277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016462.241279:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016462.241282:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334105856:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016464.278633:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016467.248760:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334105984:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016467.248824:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016467.248847:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016467/real 1734016467] req@ffff8800a78a1500 x1818247334105984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016467.248865:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334105984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016467.248872:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016467.248874:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016467.248877:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334105984:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016467.248881:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334106112:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016467.248891:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016467.248895:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016467/real 1734016467] req@ffff8800a78a1880 x1818247334106112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016467.248901:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334106112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016467.248904:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016467.248906:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016467.248908:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334106112:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016467.248911:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334106240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016467.248919:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016467.248922:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016467/real 1734016467] req@ffff8800a78a1f80 x1818247334106240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016467.248927:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334106240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016467.248931:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016467.248933:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016467.248934:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334106240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016467.248938:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334106368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016467.248946:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016467.248949:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016467/real 1734016467] req@ffff8800a78a0700 x1818247334106368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016467.248954:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334106368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016467.248957:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016467.248959:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016467.248960:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334106368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016467.248963:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334106496:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016467.248971:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016467.248974:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016467/real 1734016467] req@ffff8800a78a3b80 x1818247334106496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016467.248979:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334106496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016467.248982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016467.248983:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016467.248985:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334106496:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016467.249006:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334106624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016467.249015:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016467.249019:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016467/real 1734016467] req@ffff8800a78a2d80 x1818247334106624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016467.249025:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334106624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016467.249029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016467.249030:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016467.249031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334106624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016467.249034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334106752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016467.249041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016467.249044:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016467/real 1734016467] req@ffff8800a78a0a80 x1818247334106752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016467.249049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334106752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016467.249052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016467.249053:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016467.249055:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334106752:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016467.284717:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016470.290702:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016472.256823:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334106880:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016472.256866:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016472.256883:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016472/real 1734016472] req@ffff8800a78a2680 x1818247334106880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016472.256899:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334106880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016472.256904:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016472.256906:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016472.256909:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334106880:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016472.256912:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334107008:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016472.256922:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016472.256926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016472/real 1734016472] req@ffff8800a78a3480 x1818247334107008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016472.256931:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334107008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016472.256935:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016472.256936:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016472.256956:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334107008:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016472.256978:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334107136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016472.256993:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016472.257000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016472/real 1734016472] req@ffff8800a78a2300 x1818247334107136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016472.257010:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2300 x1818247334107136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016472.257018:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016472.257021:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016472.257024:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334107136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016472.257030:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334107264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016472.257042:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016472.257048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016472/real 1734016472] req@ffff8800a78a0a80 x1818247334107264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016472.257059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334107264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016472.257066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016472.257069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016472.257072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334107264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016472.257077:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334107392:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016472.257089:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016472.257095:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016472/real 1734016472] req@ffff8800a78a2d80 x1818247334107392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016472.257105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334107392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016472.257112:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016472.257115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016472.257118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334107392:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016472.257123:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334107520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016472.257138:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016472.257145:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016472/real 1734016472] req@ffff8800a78a3b80 x1818247334107520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016472.257155:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334107520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016472.257162:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016472.257164:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016472.257167:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334107520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016472.257173:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334107648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016472.257185:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016472.257191:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016472/real 1734016472] req@ffff8800a78a0700 x1818247334107648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016472.257201:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334107648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016472.257208:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016472.257210:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016472.257213:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334107648:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016473.296720:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016476.302830:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016477.264913:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334107776:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016477.264980:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016477.265006:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016477/real 1734016477] req@ffff8800a78a1f80 x1818247334107776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016477.265034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334107776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016477.265044:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016477.265048:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016477.265052:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334107776:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016477.265060:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334107904:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016477.265077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016477.265084:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016477/real 1734016477] req@ffff8800a78a1880 x1818247334107904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016477.265095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334107904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016477.265102:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016477.265105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016477.265108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334107904:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016477.265114:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334108032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016477.265127:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016477.265133:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016477/real 1734016477] req@ffff8800a78a1500 x1818247334108032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016477.265143:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334108032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016477.265151:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016477.265154:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016477.265157:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334108032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016477.265181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334108160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016477.265194:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016477.265200:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016477/real 1734016477] req@ffff8800a78a0700 x1818247334108160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016477.265210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334108160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016477.265218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016477.265220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016477.265223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334108160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016477.265228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334108288:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016477.265240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016477.265247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016477/real 1734016477] req@ffff8800a78a3b80 x1818247334108288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016477.265257:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334108288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016477.265263:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016477.265266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016477.265269:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334108288:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016477.265275:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334108416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016477.265290:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016477.265296:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016477/real 1734016477] req@ffff8800a78a2d80 x1818247334108416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016477.265307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334108416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016477.265313:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016477.265316:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016477.265318:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334108416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016477.265324:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334108544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016477.265336:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016477.265342:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016477/real 1734016477] req@ffff8800a78a0a80 x1818247334108544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016477.265352:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334108544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016477.265358:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016477.265360:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016477.265363:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334108544:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016479.308796:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016480.310622:0:10059:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 10/10 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1734016480.310719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016480.311172:0:1983:0:(client.c:1722:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8800a78a2300 x1818247334108672/t0(0) o103->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:17/18 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 00010000:00010000:3.0:1734016481.312854:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016482.272861:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334108800:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016482.272934:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016482.272966:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016482/real 1734016482] req@ffff8800a78a3480 x1818247334108800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016482.273005:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334108800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016482.273020:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016482.273025:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016482.273033:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334108800:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016482.273044:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334108928:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016482.273071:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016482.273082:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016482/real 1734016482] req@ffff8800a78a2680 x1818247334108928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016482.273100:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334108928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016482.273113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016482.273117:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016482.273122:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334108928:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016482.273152:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334109056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016482.273172:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016482.273182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016482/real 1734016482] req@ffff8800a78a0a80 x1818247334109056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016482.273200:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334109056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016482.273212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016482.273217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016482.273222:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334109056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016482.273231:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334109184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016482.273250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016482.273260:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016482/real 1734016482] req@ffff8800a78a2d80 x1818247334109184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016482.273295:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334109184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016482.273308:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016482.273311:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016482.273317:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334109184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016482.273326:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334109312:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016482.273353:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016482.273408:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016482/real 1734016482] req@ffff8800a78a3b80 x1818247334109312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016482.273429:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334109312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016482.273442:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016482.273446:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016482.273451:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334109312:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016482.273460:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334109440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016482.273532:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016482.273544:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016482/real 1734016482] req@ffff8800a78a0700 x1818247334109440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016482.273562:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334109440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016482.273575:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016482.273578:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016482.273583:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334109440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016482.273593:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334109568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016482.273611:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016482.273621:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016482/real 1734016482] req@ffff8800a78a1500 x1818247334109568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016482.273637:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334109568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016482.273650:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016482.273653:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016482.273658:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334109568:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016482.314729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016484.318755:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016485.320784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016487.280924:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334109696:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734016487.281011:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016487.281043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016487/real 1734016487] req@ffff8800a78a1880 x1818247334109696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016487.281074:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334109696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016487.281085:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016487.281088:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016487.281093:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334109696:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016487.281101:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334109824:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016487.281119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016487.281127:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016487/real 1734016487] req@ffff8800a78a1f80 x1818247334109824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016487.281138:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334109824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016487.281145:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016487.281148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016487.281151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334109824:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016487.281157:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334109952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016487.281171:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016487.281177:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016487/real 1734016487] req@ffff8800a78a1500 x1818247334109952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016487.281187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1500 x1818247334109952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016487.281195:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016487.281198:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016487.281201:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334109952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016487.281230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334110080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016487.281244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016487.281251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016487/real 1734016487] req@ffff8800a78a0700 x1818247334110080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016487.281262:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334110080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016487.281269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016487.281271:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016487.281274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334110080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016487.281333:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334110208:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734016487.281350:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016487.281357:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016487/real 1734016487] req@ffff8800a78a3b80 x1818247334110208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016487.281367:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334110208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016487.281375:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016487.281377:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016487.281380:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334110208:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016487.281385:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334110336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016487.281401:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016487.281408:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016487/real 1734016487] req@ffff8800a78a2d80 x1818247334110336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016487.281419:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2d80 x1818247334110336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016487.281426:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016487.281428:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016487.281431:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334110336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016487.281436:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334110464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734016487.281449:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734016487.281455:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016487/real 1734016487] req@ffff8800a78a0a80 x1818247334110464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016487.281527:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334110464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016487.281535:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016487.281538:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734016487.281542:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334110464:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016487.324771:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016488.326766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016490.330638:0:4749:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016491.332758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734016492.288812:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334110592:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734016492.288905:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334110720:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016492.288926:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334110848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016492.288948:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334110976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016492.288963:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334111104:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734016492.288978:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334111232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734016492.288993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334111360:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734016494.338704:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734016495.294698:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734016495.294742:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734016495.294760:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016495.294792:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016495.294799:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016495.294803:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016495.294807:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016495.294811:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016495.294814:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734016497.344849:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016500.350796:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016503.356659:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016506.362660:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016509.368803:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016512.374764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016515.380733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016518.386786:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016521.392797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016524.398749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016527.404774:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016530.410755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016533.416769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016536.422805:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016539.428779:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734016542.368628:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016542.368637:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734016542.368642:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016542.368652:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 809 00000100:00080000:0.0:1734016542.368661:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016542.368669:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734016542.368705:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334119552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016542.368744:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016542.368748:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016542.368750:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016542.368755:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 809 00000100:00080000:0.0:1734016542.368758:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016542.368761:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016542.368772:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016542.368782:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334119680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016542.368792:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016542.368795:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016542.368798:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016542.368802:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 809 00000100:00080000:0.0:1734016542.368805:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016542.368808:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016542.368813:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734016542.368822:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334119808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016542.368830:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016542.368833:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016542.368835:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016542.368838:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 809 00000100:00080000:0.0:1734016542.368840:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016542.368843:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016542.368848:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734016542.368858:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334119936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016542.368867:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016542.368870:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734016542.368871:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016542.368874:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 809 00000100:00080000:0.0:1734016542.368877:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016542.368880:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016542.368884:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734016542.368892:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334120064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016542.368901:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016542.368904:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734016542.368906:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016542.368909:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 809 00000100:00080000:0.0:1734016542.368912:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016542.368915:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016542.368920:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016542.368927:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334120192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016542.368934:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734016542.368937:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734016542.368939:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734016542.368942:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 809 00000100:00080000:0.0:1734016542.368945:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734016542.368948:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734016542.368952:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016542.368960:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334120320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734016542.434729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016545.440710:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016548.446725:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016551.452754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016554.458710:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016557.464753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734016560.398756:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734016560.398803:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734016560.398822:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016560.398857:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016560.398865:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016560.398872:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016560.398880:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016560.398886:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016560.398892:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734016560.470735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016563.476734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016566.482801:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016569.488771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016572.494714:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016575.500744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016578.506797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016581.512768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016584.518689:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016587.524762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016590.530783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016593.536714:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016596.542763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016599.548758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016602.554763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016605.560678:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016608.566704:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016611.572716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016614.578745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016617.584745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016619.588672:0:10059:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 14/14 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1734016619.588703:0:10059:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 3/17 locks to regular blp list, 2 blwis in pool 00010000:00010000:0.0:1734016619.588739:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 14 locks of 3 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1734016619.588749:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016619.588788:0:5248:0:(vvp_object.c:134:vvp_conf_set()) [0x200000401:0xc:0x0]: losing layout lock 00010000:00010000:3.0:1734016619.588797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016619.588819:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016619.588827:0:5248:0:(vvp_object.c:134:vvp_conf_set()) [0x200000401:0xc:0x0]: losing layout lock 00010000:00010000:0.0:1734016619.588833:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781d200/0xa50fd972cb8e7988 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x5e3a3adc27117dfc expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.588846:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364d80/0xa50fd972cb8e5f5d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27116019 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016619.588855:0:5248:0:(vvp_object.c:134:vvp_conf_set()) [0x200000401:0xf:0x0]: losing layout lock 00010000:00010000:0.0:1734016619.588861:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366400/0xa50fd972cb8e5f4f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27115fe8 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016619.588870:0:5248:0:(vvp_object.c:134:vvp_conf_set()) [0x200000401:0xe:0x0]: losing layout lock 00010000:00010000:0.0:1734016619.588876:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781e400/0xa50fd972cb8e5f10 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x5e3a3adc27115f5c expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.588894:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2640/0xa50fd972cb8e5e4c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc27115d79 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016619.588903:0:5248:0:(vvp_object.c:134:vvp_conf_set()) [0x200000401:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1734016619.588911:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898c240/0xa50fd972cb8df3ee lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x9:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc2710d324 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016619.588917:0:5248:0:(vvp_object.c:134:vvp_conf_set()) [0x200000401:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1734016619.588922:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898ed00/0xa50fd972cb8df434 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x5e3a3adc2710d3b0 expref: -99 pid: 9668 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.588940:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa3661c0/0xa50fd972cb8df42d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc2710d3a2 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016619.588949:0:5248:0:(vvp_object.c:134:vvp_conf_set()) [0x200000401:0x8:0x0]: losing layout lock 00010000:00010000:0.0:1734016619.588955:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c0d80/0xa50fd972cb8df3d9 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x8:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc2710d2e5 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016619.588961:0:5248:0:(vvp_object.c:134:vvp_conf_set()) [0x200000401:0x8:0x0]: losing layout lock 00010000:00010000:0.0:1734016619.588966:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365440/0xa50fd972cb8df3d2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x5e3a3adc2710d2d7 expref: -99 pid: 9668 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.588985:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898cb40/0xa50fd972cb8df3b6 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x7:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x5e3a3adc2710d29f expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1734016619.588992:0:5248:0:(vvp_object.c:134:vvp_conf_set()) [0x200000401:0x7:0x0]: losing layout lock 00010000:00010000:0.0:1734016619.588997:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898dd40/0xa50fd972cb8df3af lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x7:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x5e3a3adc2710d291 expref: -99 pid: 9668 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589081:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589089:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 1/0,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589096:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781d200/0xa50fd972cb8e7988 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x5e3a3adc27117dfc expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589103:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364d80/0xa50fd972cb8e5f5d lrc: 1/0,0 mode: --/PR res: [0x200000401:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27116019 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589114:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366400/0xa50fd972cb8e5f4f lrc: 1/0,0 mode: --/PR res: [0x200000401:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27115fe8 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589120:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781e400/0xa50fd972cb8e5f10 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x5e3a3adc27115f5c expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589126:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2640/0xa50fd972cb8e5e4c lrc: 1/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27115d79 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589132:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898c240/0xa50fd972cb8df3ee lrc: 1/0,0 mode: --/CR res: [0x200000401:0x9:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc2710d324 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589138:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898ed00/0xa50fd972cb8df434 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x5e3a3adc2710d3b0 expref: -99 pid: 9668 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589144:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa3661c0/0xa50fd972cb8df42d lrc: 1/0,0 mode: --/PR res: [0x200000401:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc2710d3a2 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589150:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c0d80/0xa50fd972cb8df3d9 lrc: 1/0,0 mode: --/CR res: [0x200000401:0x8:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc2710d2e5 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589156:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365440/0xa50fd972cb8df3d2 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x5e3a3adc2710d2d7 expref: -99 pid: 9668 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589163:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898cb40/0xa50fd972cb8df3b6 lrc: 1/0,0 mode: --/CR res: [0x200000401:0x7:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc2710d29f expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589169:0:5248:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898dd40/0xa50fd972cb8df3af lrc: 1/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x5e3a3adc2710d291 expref: -99 pid: 9668 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589174:0:5248:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 14 locks packed 00010000:00010000:0.0:1734016619.589209:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365680/0xa50fd972cb8e79f1 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27117e8f expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589229:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c3cc0/0xa50fd972cb8e5efb lrc: 0/0,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27115f1d expref: -99 pid: 10405 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589238:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781d200/0xa50fd972cb8e7988 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x5e3a3adc27117dfc expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589252:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa364d80/0xa50fd972cb8e5f5d lrc: 0/0,0 mode: --/PR res: [0x200000401:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27116019 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589262:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa366400/0xa50fd972cb8e5f4f lrc: 0/0,0 mode: --/PR res: [0x200000401:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27115fe8 expref: -99 pid: 10406 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589273:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a781e400/0xa50fd972cb8e5f10 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x5e3a3adc27115f5c expref: -99 pid: 10406 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589283:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c2640/0xa50fd972cb8e5e4c lrc: 0/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc27115d79 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589292:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898c240/0xa50fd972cb8df3ee lrc: 0/0,0 mode: --/CR res: [0x200000401:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc2710d324 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589300:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898ed00/0xa50fd972cb8df434 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x5e3a3adc2710d3b0 expref: -99 pid: 9668 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589311:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa3661c0/0xa50fd972cb8df42d lrc: 0/0,0 mode: --/PR res: [0x200000401:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc2710d3a2 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589325:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a78c0d80/0xa50fd972cb8df3d9 lrc: 0/0,0 mode: --/CR res: [0x200000401:0x8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc2710d2e5 expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589335:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800aa365440/0xa50fd972cb8df3d2 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x8:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x5e3a3adc2710d2d7 expref: -99 pid: 9668 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589345:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898cb40/0xa50fd972cb8df3b6 lrc: 0/0,0 mode: --/CR res: [0x200000401:0x7:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x5e3a3adc2710d29f expref: -99 pid: 9668 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1734016619.589352:0:5248:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88012a571000 lock: ffff8800a898dd40/0xa50fd972cb8df3af lrc: 0/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x5e3a3adc2710d291 expref: -99 pid: 9668 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1734016619.589363:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016619.589365:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016619.589367:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016620.590719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016622.594728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016624.598753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734016625.502745:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:0.0:1734016625.502790:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734016625.502809:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016625.502844:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016625.502851:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016625.502858:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016625.502865:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016625.502871:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016625.502877:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734016626.602735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016628.606697:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016630.610723:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016632.614745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016634.618806:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016636.622761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016638.626773:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016640.630743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016642.634715:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016644.638767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016646.642779:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016648.646735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016650.650797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016652.654754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016654.658722:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016656.662737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016658.666776:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016660.670753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016662.674766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016664.678741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016666.682737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016668.686742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016670.690761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016672.694726:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016674.698767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016676.702750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016678.706757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016680.710748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016682.714742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016684.718688:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016686.722758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016688.726735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734016690.606756:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734016690.606813:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734016690.606859:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016690.606903:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016690.606912:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016690.606919:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016690.606927:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016690.606934:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016690.606941:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734016690.730703:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016692.734759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016694.738742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016696.742800:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016698.746797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016700.750751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016702.754807:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016704.758784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016706.762781:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016708.766758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016710.770759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016712.774741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016714.778721:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016716.782741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016718.786797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016720.790741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016722.794739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016724.798734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016726.802748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016728.806745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016729.808670:0:10059:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1734016729.808800:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016729.809210:0:1984:0:(client.c:1722:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8800a78a2d80 x1818247334153728/t0(0) o103->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 00010000:00010000:3.0:1734016730.810775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016732.814732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016734.818770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016736.822776:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016738.826739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016740.830726:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016742.834734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016743.836803:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016744.838712:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016745.840749:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016746.842775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016747.844726:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016748.846767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016749.848750:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00020000:3.0:1734016750.199632:0:10406:0:(osc_cache.c:908:osc_extent_wait()) extent ffff8800b6ee7600@{[408 -> 409/1023], [3|0|+|rpc|wiuY|ffff8800a90061a0], [32768|2|+|-|ffff8800aa365200|1024|ffff8800b6d6a220]} lustre-OST0001-osc-ffff88012a571000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:3.0:1734016750.210623:0:10406:0:(osc_cache.c:908:osc_extent_wait()) ### extent: ffff8800b6ee7600 ns: lustre-OST0001-osc-ffff88012a571000 lock: ffff8800aa365200/0xa50fd972cb8e7a37 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 942080->954367) gid 0 flags: 0x800029400020000 nid: local remote: 0x5e3a3adc27117edc expref: -99 pid: 10406 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016750.850763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016751.852748:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016752.854731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016753.856716:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016754.858803:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734016755.710801:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734016755.710851:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734016755.710870:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016755.710907:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016755.710916:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016755.710923:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016755.710930:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016755.710937:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016755.710944:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734016755.860736:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016756.862734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016757.864752:0:5248:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016758.866738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1734016759.868686:0:10059:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1734016759.868713:0:10059:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:3.0:1734016759.868798:0:5248:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1734016759.868813:0:5248:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a898c6c0/0xa50fd972cb8df362 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x5e3a3adc2710d1aa expref: -99 pid: 9553 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734016759.869302:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016759.869305:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016759.869307:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016759.869506:0:1983:0:(client.c:1722:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8800a78a1500 x1818247334159232/t0(0) o103->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 00010000:00010000:3.0:1734016760.870714:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016762.874745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016764.878792:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016766.882751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016768.886683:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016770.890721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016772.894696:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016774.898765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016776.902768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016778.906738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016780.910741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016782.914740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016784.918756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016786.922655:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016788.926733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016790.930762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016792.934729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016794.938802:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016796.942706:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016798.946727:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016800.950718:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016802.954743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016804.958707:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016806.962732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016808.966743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016810.970733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016812.974692:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016814.978793:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016816.982758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016818.986733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734016820.814786:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734016820.814838:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734016820.814860:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016820.814907:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016820.814921:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016820.814932:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016820.814942:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016820.814952:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016820.814962:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734016820.990765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016822.994759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016824.998775:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016827.002754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016829.006755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016831.010768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016833.014743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016835.018731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016837.022748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016839.026784:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016841.030749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016843.034768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016845.038773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016847.042736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016849.046751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016851.050738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016853.054717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016855.058764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016857.062769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016858.064757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016859.066764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016860.068789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016861.070798:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016862.072814:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016863.074749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016864.076798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016865.078770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016866.080775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016867.082855:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016868.084756:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016869.086748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016870.088788:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016871.090790:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016872.092797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016873.094823:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016874.096789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016875.098776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016876.100870:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016877.102815:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016878.104797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016879.106809:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016880.108756:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016881.110799:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016882.112839:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016883.114792:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016884.116791:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016885.118747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734016885.918710:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:0.0:1734016885.918760:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734016885.918778:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016885.918815:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016885.918822:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016885.918829:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016885.918836:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016885.918843:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734016885.918849:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734016886.120743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016887.122760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016888.124739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016889.126752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016890.128728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016891.130698:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016892.132798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016893.134767:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016894.136762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016895.138752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016896.140716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016897.142756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016898.144716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016899.146784:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016900.148742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016901.150768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016902.152795:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016903.154730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016904.156767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016905.158700:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016906.160720:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016907.162753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016908.164735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016909.166746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016910.168742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016911.170720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016912.172746:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016913.174753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016914.176774:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016915.178747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016917.182837:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016919.186754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016921.190791:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016923.194758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016925.198765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016927.202767:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016929.206731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016931.210646:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016933.214781:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016935.218738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016937.222759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016939.226722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016941.230744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016943.234729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016945.238754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016947.242763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016949.246770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734016951.022758:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734016951.022807:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734016951.022826:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016951.022861:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016951.022869:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016951.022876:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016951.022883:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016951.022889:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734016951.022895:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734016951.250751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016953.254759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016955.258740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016957.262728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016959.266752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016961.270750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016963.274729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016965.278762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016967.282730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016969.286760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016971.290746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016973.294741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016975.298767:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016977.302740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016979.306748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016981.310729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016983.314737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016985.318727:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016987.322802:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016989.326766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016991.330752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016993.334747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016995.338727:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016997.342807:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016999.346762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017001.350769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017003.354708:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017005.358729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017007.362746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017009.366760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017011.370715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017013.374743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017015.378741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017016.126749:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017016.126796:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017016.126815:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017016.126851:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017016.126859:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017016.126866:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017016.126873:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017016.126879:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017016.126885:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734017017.382752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017019.386757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017021.390728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017023.394734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017025.398741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017027.402732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017029.406705:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017031.410749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017033.152633:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017033.152643:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017033.152647:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017033.152657:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1299 00000100:00080000:0.0:1734017033.152666:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017033.152673:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017033.152710:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334207744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017033.152747:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017033.152751:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017033.152753:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017033.152757:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1299 00000100:00080000:0.0:1734017033.152761:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017033.152765:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017033.152775:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017033.152786:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334207872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017033.152796:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017033.152800:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017033.152801:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017033.152805:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1299 00000100:00080000:0.0:1734017033.152808:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017033.152811:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017033.152816:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017033.152824:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334208000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017033.152833:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017033.152836:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017033.152838:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017033.152841:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1299 00000100:00080000:0.0:1734017033.152843:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017033.152846:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017033.152850:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017033.152859:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334208128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017033.152868:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017033.152870:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017033.152872:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017033.152875:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1299 00000100:00080000:0.0:1734017033.152878:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017033.152882:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017033.152886:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017033.152893:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334208256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017033.152901:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017033.152904:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017033.152905:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017033.152908:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1299 00000100:00080000:0.0:1734017033.152911:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017033.152914:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017033.152918:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017033.152926:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334208384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017033.152934:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017033.152937:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017033.152938:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017033.152942:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1299 00000100:00080000:0.0:1734017033.152944:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017033.152947:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017033.152951:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017033.152959:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334208512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017033.414728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017035.418747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017037.422763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017038.160606:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017038.160617:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017038.160622:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017038.160633:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1304 00000100:00080000:0.0:1734017038.160641:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017038.160649:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017038.160687:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334208640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017038.160722:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017038.160726:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017038.160728:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017038.160733:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1304 00000100:00080000:0.0:1734017038.160736:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017038.160740:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017038.160750:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017038.160761:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334208768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017038.160771:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017038.160774:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017038.160776:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017038.160780:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1304 00000100:00080000:0.0:1734017038.160783:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017038.160786:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017038.160792:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017038.160801:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334208896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017038.160810:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017038.160813:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017038.160815:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017038.160818:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1304 00000100:00080000:0.0:1734017038.160820:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017038.160823:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017038.160827:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017038.160835:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334209024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017038.160844:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017038.160848:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017038.160849:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017038.160852:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1304 00000100:00080000:0.0:1734017038.160856:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017038.160859:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017038.160864:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017038.160871:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334209152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017038.160880:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017038.160882:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017038.160885:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017038.160888:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1304 00000100:00080000:0.0:1734017038.160891:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017038.160895:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017038.160936:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017038.160947:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334209280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017038.160956:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017038.160959:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017038.160961:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017038.160964:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1304 00000100:00080000:0.0:1734017038.160968:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017038.160971:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017038.160975:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017038.160983:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334209408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017039.426750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017041.430725:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017043.434737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017045.438720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017047.442809:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017049.446784:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017051.450739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017053.454765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017055.458763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017057.462721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017059.466695:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017061.470763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017063.200721:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017063.200734:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017063.200739:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017063.200753:0:9727:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1329 00000100:00080000:0.0:1734017063.200764:0:9727:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017063.200772:0:9727:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017063.200818:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334213120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017063.200859:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017063.200864:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017063.200866:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017063.200871:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1329 00000100:00080000:0.0:1734017063.200874:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017063.200878:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017063.200889:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017063.200901:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334213248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017063.200911:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017063.200914:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017063.200916:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017063.200920:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1329 00000100:00080000:0.0:1734017063.200923:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017063.200926:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017063.200931:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017063.200939:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334213376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017063.200948:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017063.200951:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017063.200953:0:9727:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017063.200956:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1329 00000100:00080000:0.0:1734017063.200959:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017063.200963:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017063.200967:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017063.200976:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334213504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017063.200984:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017063.200987:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017063.200988:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017063.200992:0:9727:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1329 00000100:00080000:0.0:1734017063.200995:0:9727:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017063.200998:0:9727:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017063.201002:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017063.201010:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334213632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017063.201018:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017063.201021:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017063.201022:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017063.201026:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1329 00000100:00080000:0.0:1734017063.201029:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017063.201032:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017063.201036:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017063.201046:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334213760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017063.201054:0:9727:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017063.201058:0:9727:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017063.201060:0:9727:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017063.201064:0:9727:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1329 00000100:00080000:0.0:1734017063.201067:0:9727:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017063.201070:0:9727:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017063.201074:0:9727:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017063.201082:0:9727:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334213888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017063.474726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017065.478743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017067.482775:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017068.208591:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017068.208602:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017068.208606:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017068.208617:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1334 00000100:00080000:0.0:1734017068.208626:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017068.208634:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017068.208671:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334214016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017068.208707:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017068.208711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017068.208713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017068.208718:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1334 00000100:00080000:0.0:1734017068.208721:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017068.208724:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017068.208735:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017068.208746:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334214144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017068.208757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017068.208760:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017068.208761:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017068.208765:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1334 00000100:00080000:0.0:1734017068.208768:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017068.208771:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017068.208776:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017068.208786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334214272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017068.208795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017068.208798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017068.208800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017068.208803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1334 00000100:00080000:0.0:1734017068.208805:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017068.208809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017068.208813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017068.208821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334214400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017068.208830:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017068.208833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017068.208835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017068.208838:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1334 00000100:00080000:0.0:1734017068.208842:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017068.208845:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017068.208849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017068.208857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334214528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017068.208865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017068.208868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017068.208869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017068.208872:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1334 00000100:00080000:0.0:1734017068.208875:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017068.208878:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017068.208882:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017068.208890:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334214656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017068.208897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017068.208900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017068.208902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017068.208905:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1334 00000100:00080000:0.0:1734017068.208908:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017068.208911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017068.208914:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017068.208921:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334214784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017069.486722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017071.490777:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734017073.216805:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334214912:192.168.201.116@tcp:250:kworker.0 00000800:00000100:0.0:1734017073.216870:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734017073.216895:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017073/real 1734017073] req@ffff8800a78a1f80 x1818247334214912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017073.216915:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334214912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017073.216926:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017073.216929:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734017073.216934:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334214912:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734017073.216957:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334215040:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734017073.216975:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734017073.216983:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017073/real 1734017073] req@ffff8800a78a3b80 x1818247334215040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017073.216995:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334215040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017073.217003:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017073.217005:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734017073.217009:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334215040:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734017073.217015:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334215168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734017073.217029:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734017073.217035:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017073/real 1734017073] req@ffff8800a78a0a80 x1818247334215168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017073.217046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334215168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017073.217054:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017073.217057:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734017073.217060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334215168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734017073.217065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334215296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734017073.217078:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734017073.217084:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017073/real 1734017073] req@ffff8800a78a2680 x1818247334215296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017073.217094:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334215296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017073.217101:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017073.217103:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734017073.217106:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334215296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734017073.217112:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334215424:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734017073.217124:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734017073.217130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017073/real 1734017073] req@ffff8800a78a0700 x1818247334215424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017073.217140:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334215424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017073.217148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017073.217150:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734017073.217153:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334215424:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734017073.217170:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334215552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734017073.217183:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734017073.217189:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017073/real 1734017073] req@ffff8800a78a1880 x1818247334215552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017073.217200:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334215552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017073.217207:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017073.217210:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734017073.217213:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334215552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734017073.217219:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334215680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734017073.217231:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734017073.217237:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017073/real 1734017073] req@ffff8800a78a3480 x1818247334215680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017073.217247:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334215680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017073.217254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017073.217256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734017073.217259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334215680:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017073.494738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017075.498719:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017077.502754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734017078.224810:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334215808:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734017078.224907:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334215936:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734017078.224928:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334216064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734017078.224952:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334216192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734017078.224967:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334216320:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734017078.224981:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334216448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734017078.224996:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334216576:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017079.506773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017081.230853:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017081.230922:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017081.230948:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017081.231001:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017081.231011:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017081.231018:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017081.231026:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017081.231034:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017081.231040:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734017081.510742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017083.514747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017085.518763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017087.522750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017089.526764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017091.530776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017093.534759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017095.538749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017097.542837:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017099.546773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017101.550754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017103.554778:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017105.558721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017107.562764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017109.566798:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017111.570748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017113.574795:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017115.578783:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017117.582752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017118.288734:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017118.288746:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017118.288752:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017118.288765:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1384 00000100:00080000:0.0:1734017118.288776:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017118.288785:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017118.288831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334222976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017118.288874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017118.288879:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017118.288881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017118.288886:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1384 00000100:00080000:0.0:1734017118.288889:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017118.288892:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017118.288904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017118.288917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334223104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017118.288927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017118.288931:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017118.288933:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017118.288937:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1384 00000100:00080000:0.0:1734017118.288940:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017118.288943:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017118.288948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017118.288957:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334223232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017118.288966:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017118.288970:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017118.288971:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017118.288975:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1384 00000100:00080000:0.0:1734017118.288977:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017118.288981:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017118.288985:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017118.288993:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334223360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017118.289004:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017118.289007:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017118.289009:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017118.289014:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1384 00000100:00080000:0.0:1734017118.289017:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017118.289021:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017118.289025:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017118.289033:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334223488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017118.289041:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017118.289044:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017118.289046:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017118.289050:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1384 00000100:00080000:0.0:1734017118.289052:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017118.289056:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017118.289060:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017118.289070:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334223616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017118.289078:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017118.289081:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017118.289082:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017118.289086:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1384 00000100:00080000:0.0:1734017118.289089:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017118.289093:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017118.289097:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017118.289105:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334223744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017119.586756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017121.590719:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017123.296602:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017123.296613:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017123.296618:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017123.296629:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1389 00000100:00080000:0.0:1734017123.296638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017123.296645:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017123.296683:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334223872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017123.296717:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017123.296721:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017123.296723:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017123.296728:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1389 00000100:00080000:0.0:1734017123.296731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017123.296735:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017123.296745:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017123.296755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334224000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017123.296765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017123.296768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017123.296770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017123.296774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1389 00000100:00080000:0.0:1734017123.296777:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017123.296780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017123.296786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017123.296796:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334224128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017123.296805:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017123.296808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017123.296809:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017123.296812:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1389 00000100:00080000:0.0:1734017123.296815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017123.296818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017123.296822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017123.296830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334224256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017123.296839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017123.296842:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017123.296843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017123.296847:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1389 00000100:00080000:0.0:1734017123.296849:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017123.296852:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017123.296856:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017123.296863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334224384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017123.296872:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017123.296875:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017123.296877:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017123.296881:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1389 00000100:00080000:0.0:1734017123.296884:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017123.296887:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017123.296891:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017123.296899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334224512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017123.296906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017123.296909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017123.296911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017123.296914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1389 00000100:00080000:0.0:1734017123.296916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017123.296920:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017123.296924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017123.296931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334224640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017123.594739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017125.598750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017127.602724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017128.304613:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017128.304624:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017128.304629:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017128.304642:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1394 00000100:00080000:0.0:1734017128.304653:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017128.304662:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017128.304711:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334224768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017128.304758:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017128.304762:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017128.304764:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017128.304769:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1394 00000100:00080000:0.0:1734017128.304773:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017128.304777:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017128.304788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017128.304800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334224896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017128.304810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017128.304813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017128.304814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017128.304818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1394 00000100:00080000:0.0:1734017128.304821:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017128.304825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017128.304829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017128.304839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334225024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017128.304848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017128.304851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017128.304852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017128.304856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1394 00000100:00080000:0.0:1734017128.304859:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017128.304863:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017128.304867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017128.304875:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334225152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017128.304885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017128.304888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017128.304889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017128.304893:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1394 00000100:00080000:0.0:1734017128.304895:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017128.304899:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017128.304903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017128.304911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334225280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017128.304919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017128.304922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017128.304923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017128.304927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1394 00000100:00080000:0.0:1734017128.304930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017128.304933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017128.304938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017128.304948:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334225408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017128.304956:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017128.304960:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017128.304962:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017128.304966:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1394 00000100:00080000:0.0:1734017128.304970:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017128.304973:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017128.304977:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017128.304985:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334225536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017129.606733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017131.610766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017133.614763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017135.618761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017137.622751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017139.626786:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017141.630732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017143.634769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017145.638753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017147.642716:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017148.336729:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017148.336743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017148.336749:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017148.336763:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1414 00000100:00080000:0.0:1734017148.336775:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017148.336785:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017148.336833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334228352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017148.336877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017148.336881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017148.336883:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017148.336888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1414 00000100:00080000:0.0:1734017148.336891:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017148.336895:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017148.336907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017148.336919:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334228480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017148.336930:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017148.336933:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017148.336935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017148.336939:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1414 00000100:00080000:0.0:1734017148.336941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017148.336945:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017148.336950:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017148.336961:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334228608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017148.336970:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017148.336973:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017148.336975:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017148.336978:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1414 00000100:00080000:0.0:1734017148.336981:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017148.336984:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017148.336988:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017148.336998:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334228736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017148.337008:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017148.337011:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017148.337013:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017148.337017:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1414 00000100:00080000:0.0:1734017148.337020:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017148.337058:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017148.337063:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017148.337074:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334228864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017148.337083:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017148.337087:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017148.337088:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017148.337092:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1414 00000100:00080000:0.0:1734017148.337096:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017148.337100:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017148.337104:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017148.337113:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334228992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017148.337122:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017148.337125:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017148.337127:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017148.337130:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1414 00000100:00080000:0.0:1734017148.337133:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017148.337137:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017148.337141:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017148.337149:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334229120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017149.646703:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017151.650707:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017153.344583:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017153.344594:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017153.344598:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017153.344609:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1419 00000100:00080000:0.0:1734017153.344618:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017153.344626:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017153.344662:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334229248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017153.344697:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017153.344701:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017153.344703:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017153.344707:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1419 00000100:00080000:0.0:1734017153.344711:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017153.344715:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017153.344725:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017153.344736:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334229376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017153.344746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017153.344749:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017153.344751:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017153.344755:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1419 00000100:00080000:0.0:1734017153.344758:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017153.344761:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017153.344767:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017153.344775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334229504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017153.344785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017153.344788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017153.344789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017153.344792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1419 00000100:00080000:0.0:1734017153.344795:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017153.344798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017153.344802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017153.344810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334229632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017153.344819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017153.344822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017153.344823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017153.344826:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1419 00000100:00080000:0.0:1734017153.344829:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017153.344832:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017153.344836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017153.344843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334229760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017153.344851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017153.344855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017153.344857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017153.344860:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1419 00000100:00080000:0.0:1734017153.344863:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017153.344867:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017153.344871:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017153.344880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334229888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017153.344888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017153.344891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017153.344893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017153.344896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1419 00000100:00080000:0.0:1734017153.344899:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017153.344902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017153.344906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017153.344913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334230016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017153.654772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017155.658755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017157.662700:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017158.352591:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017158.352603:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017158.352608:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017158.352618:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1424 00000100:00080000:0.0:1734017158.352628:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017158.352636:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017158.352674:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334230144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017158.352716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017158.352720:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017158.352722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017158.352727:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1424 00000100:00080000:0.0:1734017158.352730:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017158.352734:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017158.352744:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017158.352755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334230272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017158.352764:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017158.352767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017158.352769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017158.352773:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1424 00000100:00080000:0.0:1734017158.352775:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017158.352779:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017158.352784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017158.352793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334230400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017158.352801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017158.352804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017158.352806:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017158.352809:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1424 00000100:00080000:0.0:1734017158.352812:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017158.352816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017158.352820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017158.352828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334230528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017158.352837:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017158.352840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017158.352841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017158.352844:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1424 00000100:00080000:0.0:1734017158.352847:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017158.352850:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017158.352889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017158.352899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334230656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017158.352908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017158.352911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017158.352912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017158.352916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1424 00000100:00080000:0.0:1734017158.352919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017158.352922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017158.352926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017158.352934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334230784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017158.352942:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017158.352946:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017158.352947:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017158.352951:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1424 00000100:00080000:0.0:1734017158.352954:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017158.352957:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017158.352961:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017158.352968:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334230912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017159.666753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017161.670773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017163.360586:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017163.360596:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017163.360601:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017163.360612:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1429 00000100:00080000:0.0:1734017163.360621:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017163.360628:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017163.360665:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334231040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017163.360700:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017163.360704:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017163.360706:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017163.360710:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1429 00000100:00080000:0.0:1734017163.360713:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017163.360717:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017163.360728:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017163.360739:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334231168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017163.360757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017163.360760:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017163.360762:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017163.360792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1429 00000100:00080000:0.0:1734017163.360796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017163.360800:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017163.360805:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017163.360815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334231296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017163.360825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017163.360829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017163.360830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017163.360834:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1429 00000100:00080000:0.0:1734017163.360836:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017163.360839:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017163.360843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017163.360851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334231424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017163.360860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017163.360862:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017163.360864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017163.360867:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1429 00000100:00080000:0.0:1734017163.360870:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017163.360874:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017163.360877:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017163.360885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334231552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017163.360892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017163.360895:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017163.360897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017163.360900:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1429 00000100:00080000:0.0:1734017163.360903:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017163.360906:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017163.360910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017163.360919:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334231680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017163.360927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017163.360930:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017163.360931:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017163.360935:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1429 00000100:00080000:0.0:1734017163.360937:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017163.360940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017163.360944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017163.360952:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334231808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017163.674819:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017165.678812:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017167.682776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017168.368618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017168.368629:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017168.368634:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017168.368645:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1435 00000100:00080000:0.0:1734017168.368655:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017168.368663:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017168.368732:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334231936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017168.368766:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017168.368770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017168.368771:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017168.368776:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1435 00000100:00080000:0.0:1734017168.368779:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017168.368783:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017168.368793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017168.368805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334232064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017168.368815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017168.368818:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017168.368820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017168.368824:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1435 00000100:00080000:0.0:1734017168.368826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017168.368830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017168.368836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017168.368846:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334232192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017168.368856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017168.368859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017168.368861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017168.368865:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1435 00000100:00080000:0.0:1734017168.368868:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017168.368871:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017168.368875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017168.368883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334232320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017168.368891:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017168.368894:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017168.368895:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017168.368898:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1435 00000100:00080000:0.0:1734017168.368902:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017168.368905:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017168.368908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017168.368916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334232448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017168.368924:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017168.368927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017168.368929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017168.368933:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1435 00000100:00080000:0.0:1734017168.368936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017168.368940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017168.368944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017168.368952:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334232576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017168.368959:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017168.368962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017168.368964:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017168.368967:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1435 00000100:00080000:0.0:1734017168.368969:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017168.368972:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017168.368976:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017168.368984:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334232704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017169.686752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017171.690815:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017173.376657:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017173.376668:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017173.376673:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017173.376685:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1440 00000100:00080000:0.0:1734017173.376694:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017173.376701:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017173.376740:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334232832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017173.376783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017173.376787:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017173.376789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017173.376793:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1440 00000100:00080000:0.0:1734017173.376797:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017173.376801:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017173.376811:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017173.376822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334232960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017173.376832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017173.376835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017173.376836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017173.376841:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1440 00000100:00080000:0.0:1734017173.376843:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017173.376847:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017173.376852:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017173.376861:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334233088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017173.376870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017173.376873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017173.376874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017173.376877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1440 00000100:00080000:0.0:1734017173.376880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017173.376883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017173.376888:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017173.376896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334233216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017173.376904:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017173.376907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017173.376908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017173.376912:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1440 00000100:00080000:0.0:1734017173.376916:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017173.376919:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017173.376923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017173.376931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334233344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017173.376939:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017173.376942:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017173.376943:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017173.376946:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1440 00000100:00080000:0.0:1734017173.376949:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017173.376953:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017173.376957:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017173.376965:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334233472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017173.376973:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017173.376977:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017173.376978:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017173.376982:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1440 00000100:00080000:0.0:1734017173.376984:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017173.376987:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017173.376991:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017173.376999:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334233600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017173.694774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017175.698772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017177.702773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017178.384677:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017178.384689:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017178.384694:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017178.384708:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1445 00000100:00080000:0.0:1734017178.384719:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017178.384729:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017178.384777:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334233728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017178.384812:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017178.384817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017178.384819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017178.384824:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1445 00000100:00080000:0.0:1734017178.384827:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017178.384831:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017178.384841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017178.384852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334233856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017178.384864:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017178.384868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017178.384869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017178.384873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1445 00000100:00080000:0.0:1734017178.384876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017178.384879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017178.384884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017178.384894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334233984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017178.384903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017178.384907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017178.384908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017178.384912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1445 00000100:00080000:0.0:1734017178.384914:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017178.384918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017178.384922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017178.384932:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334234112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017178.384941:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017178.384944:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017178.384946:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017178.384949:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1445 00000100:00080000:0.0:1734017178.384953:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017178.384957:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017178.384961:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017178.384970:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334234240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017178.384978:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017178.384981:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017178.384983:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017178.384986:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1445 00000100:00080000:0.0:1734017178.384989:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017178.384992:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017178.384997:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017178.385006:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334234368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017178.385014:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017178.385017:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017178.385019:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017178.385022:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1445 00000100:00080000:0.0:1734017178.385025:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017178.385029:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017178.385033:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017178.385041:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334234496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017179.706698:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017181.710748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017183.392631:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017183.392642:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017183.392647:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017183.392658:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1450 00000100:00080000:0.0:1734017183.392667:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017183.392674:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017183.392711:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334234624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017183.392745:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017183.392750:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017183.392752:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017183.392756:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1450 00000100:00080000:0.0:1734017183.392759:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017183.392763:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017183.392774:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017183.392785:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334234752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017183.392795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017183.392798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017183.392800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017183.392804:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1450 00000100:00080000:0.0:1734017183.392807:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017183.392810:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017183.392816:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017183.392825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334234880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017183.392834:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017183.392837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017183.392839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017183.392842:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1450 00000100:00080000:0.0:1734017183.392844:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017183.392847:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017183.392851:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017183.392859:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334235008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017183.392868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017183.392871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017183.392872:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017183.392876:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1450 00000100:00080000:0.0:1734017183.392878:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017183.392881:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017183.392885:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017183.392892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334235136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017183.392901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017183.392904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017183.392906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017183.392909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1450 00000100:00080000:0.0:1734017183.392912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017183.392916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017183.392920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017183.392929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334235264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017183.392937:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017183.392940:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017183.392941:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017183.392944:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1450 00000100:00080000:0.0:1734017183.392947:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017183.392950:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017183.392954:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017183.392961:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334235392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017183.714736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017185.718717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017187.722769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017188.400587:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017188.400599:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017188.400604:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017188.400618:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1455 00000100:00080000:0.0:1734017188.400629:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017188.400638:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017188.400685:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334235520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017188.400732:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017188.400737:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017188.400739:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017188.400744:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1455 00000100:00080000:0.0:1734017188.400747:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017188.400751:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017188.400762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017188.400774:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334235648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017188.400783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017188.400786:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017188.400788:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017188.400792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1455 00000100:00080000:0.0:1734017188.400795:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017188.400799:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017188.400804:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017188.400815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334235776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017188.400824:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017188.400827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017188.400829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017188.400832:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1455 00000100:00080000:0.0:1734017188.400835:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017188.400839:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017188.400843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017188.400854:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334235904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017188.400863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017188.400866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017188.400868:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017188.400871:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1455 00000100:00080000:0.0:1734017188.400874:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017188.400877:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017188.400881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017188.400889:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334236032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017188.400897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017188.400900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017188.400902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017188.400906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1455 00000100:00080000:0.0:1734017188.400919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017188.400923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017188.400927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017188.400937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334236160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017188.400946:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017188.400950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017188.400952:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017188.400956:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1455 00000100:00080000:0.0:1734017188.400959:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017188.400963:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017188.400967:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017188.400975:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334236288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017189.726753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017191.730724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017193.734728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017195.738705:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017197.742756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017199.746715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017201.750735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017203.754697:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017205.758751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017207.762733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017209.766751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017211.438778:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017211.438839:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017211.438862:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017211.438911:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017211.438923:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017211.438934:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017211.438945:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017211.438987:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017211.438998:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734017211.770734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017213.774741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017215.778706:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017217.782678:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017219.786743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017221.790745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017223.794772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017225.798754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017227.802758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017229.806732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017231.810767:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017233.814735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017235.818751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017237.822730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017239.826768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017241.830763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017243.834707:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017245.838721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017247.842754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017249.846747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017251.850735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017253.854667:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017255.858721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017257.862739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017259.866715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017261.870740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017263.874725:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017265.878735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017267.882718:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017269.886776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017271.890737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017273.894722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017275.898741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017276.542772:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017276.542828:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017276.542877:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017276.542927:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017276.542941:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017276.542952:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017276.542963:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017276.542975:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017276.542986:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734017277.902762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017279.906733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017281.910774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017283.914710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017285.918746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017287.922746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017289.926793:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017291.930796:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017293.934769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017295.938756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017297.942704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017299.946740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017301.950729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017303.954691:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017305.958713:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017307.962742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017309.966745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017311.970731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017313.974715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017315.978730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017317.982755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017319.986788:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017321.990757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017323.994731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017325.998742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017328.002753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017330.006774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017332.010784:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017334.014724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017336.018776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017338.022761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017340.026748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017341.646825:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017341.646885:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017341.646911:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017341.646954:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017341.646964:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017341.646972:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017341.646979:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017341.646986:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017341.646993:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734017342.030711:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017344.034777:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017346.038736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017348.042789:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017350.046706:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017352.050737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017354.054768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017356.058753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017358.062795:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00020000:3.0:1734017359.868657:0:5248:0:(osc_cache.c:908:osc_extent_wait()) extent ffff8800aa31ea80@{[0 -> 0/1023], [3|0|+|rpc|wiuY|ffff8800a9006340], [28672|1|+|-|ffff8800a898c6c0|1024|ffff8800a9e90000]} lustre-OST0000-osc-ffff88012a571000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:3.0:1734017359.880048:0:5248:0:(osc_cache.c:908:osc_extent_wait()) ### extent: ffff8800aa31ea80 ns: lustre-OST0000-osc-ffff88012a571000 lock: ffff8800a898c6c0/0xa50fd972cb8df362 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400020000 nid: local remote: 0x5e3a3adc2710d1aa expref: -99 pid: 9553 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734017360.066723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017362.070796:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017364.074766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017366.078741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017368.082656:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017370.086727:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017372.090734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017374.094761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017376.098737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017378.102745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017380.106728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017382.110770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017384.114710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017386.118765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017388.122769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017390.126743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017392.130763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017394.134750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017396.138737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017398.142847:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017398.736800:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1665 00000100:00080000:2.0:1734017398.736813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017398.736817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017398.736822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017398.736833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334273408/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017398.736842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017398.736846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017398.736848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017398.736851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1665 00000100:00080000:2.0:1734017398.736854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017398.736857:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017398.736861:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017398.736870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334273536/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017398.736879:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017398.736882:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017398.736884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017398.736887:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1665 00000100:00080000:2.0:1734017398.736890:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017398.736893:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017398.736897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017398.736905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334273664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017398.736919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017398.736923:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017398.736924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017398.736928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1665 00000100:00080000:2.0:1734017398.736931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017398.736934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017398.736938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017398.736946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334273792/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017398.736954:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017398.736957:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017398.736959:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017398.736962:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1665 00000100:00080000:2.0:1734017398.736965:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017398.736968:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017398.736972:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017398.736980:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334273920/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734017399.144796:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017400.146610:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017401.148789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017402.150811:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017403.152786:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017403.744617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017403.744631:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017403.744636:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017403.744647:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1670 00000100:00080000:2.0:1734017403.744656:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017403.744664:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017403.744750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334274048/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017403.744783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017403.744788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017403.744789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017403.744794:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1670 00000100:00080000:2.0:1734017403.744798:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017403.744802:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017403.744813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017403.744824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334274176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017403.744835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017403.744838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017403.744840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017403.744843:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1670 00000100:00080000:2.0:1734017403.744846:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017403.744850:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017403.744854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017403.744863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334274304/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017403.744879:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017403.744883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017403.744885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017403.744888:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1670 00000100:00080000:2.0:1734017403.744890:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017403.744893:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017403.744897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017403.744905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334274432/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017403.744913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017403.744916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017403.744917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017403.744921:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1670 00000100:00080000:2.0:1734017403.744923:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017403.744926:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017403.744930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017403.744937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334274560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017403.744947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017403.744950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017403.744951:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017403.744955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1670 00000100:00080000:2.0:1734017403.744958:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017403.744961:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017403.744965:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017403.744973:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334274688/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017403.744981:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017403.744984:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017403.744986:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017403.744989:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1670 00000100:00080000:2.0:1734017403.744991:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017403.744995:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017403.744998:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017403.745006:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334274816/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017404.154753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017405.156785:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017406.158825:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017406.750929:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:0.0:1734017406.750983:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017406.751002:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017406.751042:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017406.751050:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017406.751057:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017406.751064:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017406.751071:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017406.751077:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:2.0:1734017407.160784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017408.162735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017408.752574:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017408.752585:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017408.752618:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017408.752629:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1675 00000100:00080000:2.0:1734017408.752638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017408.752646:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017408.752686:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334274944/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017408.752729:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017408.752733:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017408.752735:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017408.752739:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1675 00000100:00080000:2.0:1734017408.752743:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017408.752747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017408.752757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017408.752770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334275072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017408.752779:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017408.752782:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017408.752784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017408.752788:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1675 00000100:00080000:2.0:1734017408.752791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017408.752794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017408.752799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017408.752808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334275200/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017408.752816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017408.752819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017408.752821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017408.752824:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1675 00000100:00080000:2.0:1734017408.752826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017408.752830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017408.752834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017408.752842:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334275328/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017408.752851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017408.752855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017408.752856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017408.752860:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1675 00000100:00080000:2.0:1734017408.752862:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017408.752866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017408.752870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017408.752878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334275456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017408.752886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017408.752889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017408.752890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017408.752894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1675 00000100:00080000:2.0:1734017408.752896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017408.752899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017408.752903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017408.752911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334275584/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017408.752918:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017408.752922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017408.752923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017408.752927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1675 00000100:00080000:2.0:1734017408.752929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017408.752932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017408.752936:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017408.752943:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334275712/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734017409.164810:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017410.166768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017411.168793:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017412.170774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017413.172790:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017413.760587:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017413.760598:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017413.760603:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017413.760614:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1680 00000100:00080000:2.0:1734017413.760623:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017413.760631:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017413.760671:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334275840/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017413.760705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017413.760709:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017413.760711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017413.760716:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1680 00000100:00080000:2.0:1734017413.760720:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017413.760724:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017413.760734:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017413.760746:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334275968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017413.760756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017413.760759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017413.760761:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017413.760765:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1680 00000100:00080000:2.0:1734017413.760767:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017413.760771:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017413.760776:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017413.760785:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334276096/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017413.760794:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017413.760797:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017413.760799:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017413.760802:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1680 00000100:00080000:2.0:1734017413.760805:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017413.760808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017413.760812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017413.760821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334276224/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017413.760835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017413.760838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017413.760840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017413.760843:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1680 00000100:00080000:2.0:1734017413.760846:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017413.760849:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017413.760853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017413.760860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334276352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017413.760868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017413.760871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017413.760873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017413.760876:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1680 00000100:00080000:2.0:1734017413.760879:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017413.760882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017413.760886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017413.760893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334276480/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017413.760901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017413.760904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017413.760905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017413.760909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1680 00000100:00080000:2.0:1734017413.760911:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017413.760915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017413.760919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017413.760927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334276608/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734017414.174706:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017415.176776:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017416.178717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017417.180739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017418.182740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017418.768604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017418.768614:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017418.768618:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017418.768628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1685 00000100:00080000:2.0:1734017418.768637:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017418.768644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017418.768682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334276736/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017418.768716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017418.768720:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017418.768722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017418.768726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1685 00000100:00080000:2.0:1734017418.768729:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017418.768733:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017418.768743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017418.768755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334276864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017418.768765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017418.768768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017418.768770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017418.768774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1685 00000100:00080000:2.0:1734017418.768776:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017418.768780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017418.768784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017418.768793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334276992/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017418.768801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017418.768804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017418.768806:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017418.768809:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1685 00000100:00080000:2.0:1734017418.768811:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017418.768814:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017418.768818:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017418.768827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334277120/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017418.768836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017418.768839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017418.768841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017418.768844:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1685 00000100:00080000:2.0:1734017418.768847:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017418.768850:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017418.768854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017418.768862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334277248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017418.768870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017418.768872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017418.768874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017418.768877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1685 00000100:00080000:2.0:1734017418.768881:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017418.768884:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017418.768888:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017418.768895:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334277376/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017418.768903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017418.768906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017418.768907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017418.768910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1685 00000100:00080000:2.0:1734017418.768913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017418.768916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017418.768920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017418.768928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334277504/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734017419.184709:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017420.186694:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017421.188713:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017422.190731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017423.192777:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017423.776623:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017423.776636:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017423.776641:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017423.776651:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1690 00000100:00080000:2.0:1734017423.776660:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017423.776668:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017423.776707:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334277632/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017423.776741:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017423.776746:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017423.776748:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017423.776752:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1690 00000100:00080000:2.0:1734017423.776755:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017423.776759:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017423.776770:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017423.776781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334277760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017423.776791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017423.776794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017423.776796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017423.776800:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1690 00000100:00080000:2.0:1734017423.776803:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017423.776807:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017423.776811:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017423.776820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334277888/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017423.776829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017423.776832:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017423.776834:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017423.776837:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1690 00000100:00080000:2.0:1734017423.776839:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017423.776842:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017423.776846:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017423.776854:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334278016/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017423.776863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017423.776866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017423.776867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017423.776871:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1690 00000100:00080000:2.0:1734017423.776873:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017423.776876:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017423.776880:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017423.776888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334278144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017423.776896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017423.776899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017423.776901:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017423.776904:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1690 00000100:00080000:2.0:1734017423.776909:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017423.776912:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017423.776916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017423.776923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334278272/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017423.776931:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017423.776934:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017423.776935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017423.776939:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1690 00000100:00080000:2.0:1734017423.776941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017423.776944:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017423.776948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017423.776955:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334278400/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017424.194701:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017425.196743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017426.198763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017427.200747:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017428.202777:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017428.784616:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017428.784625:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017428.784630:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017428.784640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1695 00000100:00080000:2.0:1734017428.784649:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017428.784657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017428.784693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334278528/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017428.784728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017428.784732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017428.784733:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017428.784738:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1695 00000100:00080000:2.0:1734017428.784741:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017428.784745:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017428.784755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017428.784767:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334278656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017428.784777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017428.784780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017428.784782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017428.784786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1695 00000100:00080000:2.0:1734017428.784789:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017428.784793:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017428.784798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017428.784807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334278784/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017428.784816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017428.784819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017428.784820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017428.784823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1695 00000100:00080000:2.0:1734017428.784826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017428.784830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017428.784834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017428.784842:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334278912/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017428.784851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017428.784854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017428.784855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017428.784858:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1695 00000100:00080000:2.0:1734017428.784861:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017428.784864:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017428.784868:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017428.784875:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334279040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017428.784883:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017428.784886:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017428.784887:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017428.784891:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1695 00000100:00080000:2.0:1734017428.784893:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017428.784896:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017428.784900:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017428.784908:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334279168/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017428.784916:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017428.784920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017428.784921:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017428.784925:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1695 00000100:00080000:2.0:1734017428.784928:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017428.784931:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017428.784935:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017428.784942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334279296/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017429.204751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017430.206736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017431.208722:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017432.210724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017433.212733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017433.792618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017433.792627:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017433.792632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017433.792643:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1700 00000100:00080000:2.0:1734017433.792651:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017433.792659:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017433.792697:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334279424/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017433.792730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017433.792734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017433.792736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017433.792741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1700 00000100:00080000:2.0:1734017433.792744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017433.792748:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017433.792758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017433.792770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334279552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017433.792781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017433.792784:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017433.792786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017433.792790:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1700 00000100:00080000:2.0:1734017433.792792:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017433.792796:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017433.792800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017433.792809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334279680/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017433.792818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017433.792821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017433.792822:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017433.792826:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1700 00000100:00080000:2.0:1734017433.792829:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017433.792832:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017433.792836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017433.792843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334279808/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017433.792853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017433.792855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017433.792857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017433.792860:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1700 00000100:00080000:2.0:1734017433.792863:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017433.792866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017433.792870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017433.792877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334279936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017433.792885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017433.792888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017433.792890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017433.792893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1700 00000100:00080000:2.0:1734017433.792896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017433.792899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017433.792903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017433.792909:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334280064/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017433.792917:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017433.792920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017433.792921:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017433.792925:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1700 00000100:00080000:2.0:1734017433.792936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017433.792940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017433.792944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017433.792952:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334280192/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017434.214724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017435.216765:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017436.218738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017437.220783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017438.222748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017438.800589:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017438.800600:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017438.800605:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017438.800616:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1705 00000100:00080000:2.0:1734017438.800625:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017438.800633:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017438.800671:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334280320/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017438.800705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017438.800710:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017438.800711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017438.800716:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1705 00000100:00080000:2.0:1734017438.800719:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017438.800723:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017438.800733:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017438.800745:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334280448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017438.800755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017438.800758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017438.800760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017438.800764:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1705 00000100:00080000:2.0:1734017438.800767:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017438.800770:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017438.800776:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017438.800785:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334280576/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017438.800794:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017438.800797:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017438.800799:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017438.800802:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1705 00000100:00080000:2.0:1734017438.800804:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017438.800807:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017438.800812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017438.800820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334280704/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017438.800829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017438.800832:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017438.800833:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017438.800837:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1705 00000100:00080000:2.0:1734017438.800839:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017438.800842:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017438.800846:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017438.800854:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334280832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017438.800862:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017438.800865:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017438.800867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017438.800871:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1705 00000100:00080000:2.0:1734017438.800874:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017438.800877:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017438.800881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017438.800888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334280960/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017438.800896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017438.800899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017438.800900:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017438.800903:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1705 00000100:00080000:2.0:1734017438.800906:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017438.800909:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017438.800913:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017438.800921:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334281088/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017439.224748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017440.226707:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017441.228745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017442.230754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017443.232765:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017443.808614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017443.808623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017443.808628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017443.808639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1710 00000100:00080000:2.0:1734017443.808647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017443.808654:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017443.808693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334281216/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017443.808734:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017443.808738:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017443.808740:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017443.808745:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1710 00000100:00080000:2.0:1734017443.808748:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017443.808751:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017443.808761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017443.808773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334281344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017443.808782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017443.808785:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017443.808786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017443.808790:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1710 00000100:00080000:2.0:1734017443.808793:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017443.808797:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017443.808802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017443.808811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334281472/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017443.808819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017443.808822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017443.808824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017443.808827:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1710 00000100:00080000:2.0:1734017443.808830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017443.808834:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017443.808838:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017443.808846:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334281600/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017443.808855:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017443.808858:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017443.808860:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017443.808863:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1710 00000100:00080000:2.0:1734017443.808865:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017443.808868:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017443.808872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017443.808879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334281728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017443.808887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017443.808890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017443.808891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017443.808895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1710 00000100:00080000:2.0:1734017443.808897:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017443.808900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017443.808904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017443.808911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334281856/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017443.808918:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017443.808922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017443.808924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017443.808927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1710 00000100:00080000:2.0:1734017443.808930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017443.808933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017443.808937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017443.808944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334281984/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017444.234754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017445.236757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017446.238748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017447.240760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017448.242756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017448.816566:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017448.816576:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017448.816580:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017448.816592:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1715 00000100:00080000:2.0:1734017448.816600:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017448.816608:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017448.816647:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334282112/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017448.816681:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017448.816686:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017448.816688:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017448.816692:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1715 00000100:00080000:2.0:1734017448.816695:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017448.816699:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017448.816709:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017448.816721:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334282240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017448.816731:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017448.816735:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017448.816737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017448.816741:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1715 00000100:00080000:2.0:1734017448.816743:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017448.816747:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017448.816752:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017448.816761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334282368/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017448.816770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017448.816773:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017448.816774:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017448.816778:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1715 00000100:00080000:2.0:1734017448.816780:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017448.816783:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017448.816787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017448.816795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334282496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017448.816803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017448.816806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017448.816808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017448.816811:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1715 00000100:00080000:2.0:1734017448.816814:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017448.816817:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017448.816822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017448.816829:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334282624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017448.816837:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017448.816840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017448.816841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017448.816845:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1715 00000100:00080000:2.0:1734017448.816847:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017448.816850:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017448.816854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017448.816861:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334282752/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017448.816869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017448.816872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017448.816874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017448.816877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1715 00000100:00080000:2.0:1734017448.816880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017448.816883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017448.816887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017448.816894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334282880/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017449.244788:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017450.246763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017451.248770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017452.250737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017453.252789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017453.824618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017453.824627:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017453.824632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017453.824642:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1720 00000100:00080000:2.0:1734017453.824651:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017453.824659:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017453.824695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334283008/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017453.824730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017453.824734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017453.824736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017453.824740:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1720 00000100:00080000:2.0:1734017453.824743:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017453.824747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017453.824757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017453.824769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334283136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017453.824778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017453.824781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017453.824783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017453.824787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1720 00000100:00080000:2.0:1734017453.824790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017453.824794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017453.824799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017453.824808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334283264/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017453.824816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017453.824819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017453.824847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017453.824854:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1720 00000100:00080000:2.0:1734017453.824858:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017453.824862:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017453.824867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017453.824878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334283392/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017453.824889:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017453.824892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017453.824893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017453.824897:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1720 00000100:00080000:2.0:1734017453.824899:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017453.824902:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017453.824907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017453.824914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334283520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017453.824923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017453.824926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017453.824928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017453.824932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1720 00000100:00080000:2.0:1734017453.824935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017453.824938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017453.824942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017453.824949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334283648/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017453.824957:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017453.824960:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017453.824961:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017453.824965:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1720 00000100:00080000:2.0:1734017453.824967:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017453.824970:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017453.824974:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017453.824981:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334283776/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017454.254757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017455.256780:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017456.258820:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017457.260759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017458.262819:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017458.832507:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017458.832513:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017458.832516:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017458.832522:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1725 00000100:00080000:2.0:1734017458.832528:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017458.832532:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017458.832558:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334283904/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017458.832582:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017458.832584:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017458.832585:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017458.832587:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1725 00000100:00080000:2.0:1734017458.832589:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017458.832591:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017458.832598:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017458.832604:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334284032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017458.832609:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017458.832611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017458.832612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017458.832614:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1725 00000100:00080000:2.0:1734017458.832616:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017458.832618:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017458.832620:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017458.832625:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334284160/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017458.832629:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017458.832631:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017458.832632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017458.832633:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1725 00000100:00080000:2.0:1734017458.832634:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017458.832637:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017458.832639:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017458.832643:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334284288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017458.832648:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017458.832650:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017458.832650:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017458.832652:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1725 00000100:00080000:2.0:1734017458.832653:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017458.832655:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017458.832657:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017458.832661:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334284416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017458.832665:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017458.832667:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017458.832667:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017458.832669:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1725 00000100:00080000:2.0:1734017458.832670:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017458.832672:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017458.832674:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017458.832678:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334284544/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017458.832682:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017458.832684:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017458.832685:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017458.832687:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1725 00000100:00080000:2.0:1734017458.832688:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017458.832690:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017458.832692:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017458.832696:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334284672/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017459.264819:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017460.266797:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017461.268672:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017462.270781:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017463.272766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017463.840623:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017463.840632:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017463.840637:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017463.840647:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1730 00000100:00080000:2.0:1734017463.840680:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017463.840688:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017463.840728:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334284800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017463.840761:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017463.840765:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017463.840767:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017463.840771:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1730 00000100:00080000:2.0:1734017463.840774:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017463.840778:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017463.840789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017463.840801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334284928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017463.840811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017463.840814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017463.840816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017463.840819:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1730 00000100:00080000:2.0:1734017463.840822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017463.840825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017463.840830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017463.840839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334285056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017463.840847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017463.840850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017463.840852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017463.840855:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1730 00000100:00080000:2.0:1734017463.840857:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017463.840860:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017463.840864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017463.840873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334285184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017463.840882:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017463.840885:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017463.840886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017463.840890:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1730 00000100:00080000:2.0:1734017463.840892:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017463.840896:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017463.840900:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017463.840907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334285312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017463.840915:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017463.840917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017463.840919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017463.840922:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1730 00000100:00080000:2.0:1734017463.840925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017463.840928:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017463.840932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017463.840939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334285440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017463.840947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017463.840950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017463.840951:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017463.840955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1730 00000100:00080000:2.0:1734017463.840966:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017463.840970:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017463.840974:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017463.840981:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334285568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017464.274737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017465.276718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017466.278787:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017467.280822:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017468.282646:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017468.848628:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017468.848638:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017468.848644:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017468.848657:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1735 00000100:00080000:2.0:1734017468.848668:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017468.848678:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017468.848729:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334285696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017468.848766:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017468.848770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017468.848772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017468.848777:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1735 00000100:00080000:2.0:1734017468.848780:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017468.848785:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017468.848796:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017468.848809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334285824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017468.848818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017468.848821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017468.848823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017468.848828:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1735 00000100:00080000:2.0:1734017468.848831:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017468.848834:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017468.848840:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017468.848850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334285952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017468.848860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017468.848863:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017468.848865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017468.848868:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1735 00000100:00080000:2.0:1734017468.848871:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017468.848874:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017468.848878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017468.848887:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334286080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017468.848896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017468.848899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017468.848901:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017468.848905:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1735 00000100:00080000:2.0:1734017468.848907:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017468.848911:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017468.848915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017468.848924:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334286208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017468.848933:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017468.848936:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017468.848938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017468.848942:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1735 00000100:00080000:2.0:1734017468.848945:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017468.848949:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017468.848953:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017468.848961:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334286336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017468.848969:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017468.848972:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017468.848973:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017468.848977:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1735 00000100:00080000:2.0:1734017468.848980:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017468.848983:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017468.848987:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017468.848995:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334286464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017469.284741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017470.286811:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017471.288800:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017471.854844:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017471.854904:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017471.854930:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017471.854975:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017471.854984:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017471.854992:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017471.854999:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017471.855006:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017471.855012:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00010000:00010000:3.0:1734017472.290776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017473.292801:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017473.856585:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017473.856595:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017473.856601:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017473.856613:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1740 00000100:00080000:2.0:1734017473.856625:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017473.856635:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017473.856685:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334286592/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017473.856732:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017473.856736:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017473.856738:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017473.856743:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1740 00000100:00080000:2.0:1734017473.856747:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017473.856750:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017473.856761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017473.856774:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334286720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017473.856783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017473.856786:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017473.856788:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017473.856792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1740 00000100:00080000:2.0:1734017473.856795:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017473.856799:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017473.856804:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017473.856814:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334286848/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017473.856823:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017473.856826:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017473.856828:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017473.856832:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1740 00000100:00080000:2.0:1734017473.856834:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017473.856838:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017473.856843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017473.856852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334286976/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017473.856861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017473.856864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017473.856866:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017473.856870:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1740 00000100:00080000:2.0:1734017473.856873:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017473.856876:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017473.856880:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017473.856889:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334287104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017473.856897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017473.856900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017473.856901:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017473.856905:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1740 00000100:00080000:2.0:1734017473.856907:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017473.856911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017473.856915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017473.856923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334287232/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017473.856931:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017473.856935:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017473.856937:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017473.856941:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1740 00000100:00080000:2.0:1734017473.856945:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017473.856948:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017473.856952:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017473.856960:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334287360/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:3.0:1734017474.294783:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017475.296784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017476.298786:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017477.300784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017478.302793:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017478.864617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017478.864628:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017478.864632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017478.864643:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1745 00000100:00080000:2.0:1734017478.864652:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017478.864659:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017478.864697:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334287488/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017478.864732:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017478.864736:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017478.864738:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017478.864743:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1745 00000100:00080000:2.0:1734017478.864746:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017478.864750:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017478.864760:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017478.864772:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334287616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017478.864782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017478.864785:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017478.864787:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017478.864790:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1745 00000100:00080000:2.0:1734017478.864793:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017478.864796:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017478.864801:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017478.864810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334287744/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017478.864818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017478.864821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017478.864823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017478.864826:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1745 00000100:00080000:2.0:1734017478.864829:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017478.864832:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017478.864836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017478.864844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334287872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017478.864853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017478.864856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017478.864858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017478.864861:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1745 00000100:00080000:2.0:1734017478.864864:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017478.864867:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017478.864872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017478.864879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334288000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017478.864887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017478.864890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017478.864891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017478.864894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1745 00000100:00080000:2.0:1734017478.864897:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017478.864900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017478.864904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017478.864912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334288128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017478.864919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017478.864922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017478.864924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017478.864927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1745 00000100:00080000:2.0:1734017478.864929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017478.864933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017478.864937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017478.864944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334288256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017478.865069:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017478.865072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017478.865076:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334287744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017478.865104:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334287872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017478.865117:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017478.865123:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017478/real 1734017478] req@ffff8800a78a0700 x1818247334287872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017478.865133:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334287872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017478.865140:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017478.865143:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017478.865145:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334287872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017478.865151:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334288000:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017478.865163:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017478.865169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017478/real 1734017478] req@ffff8800a78a2680 x1818247334288000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017478.865179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334288000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017478.865186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017478.865188:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017478.865191:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334288000:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017478.865197:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334288128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017478.865217:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017478.865224:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017478/real 1734017478] req@ffff8800a78a0a80 x1818247334288128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017478.865234:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334288128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017478.865241:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017478.865243:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017478.865246:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334288128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017478.865252:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334288256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017478.865264:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017478.865270:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017478/real 1734017478] req@ffff8800a78a1f80 x1818247334288256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017478.865280:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334288256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017478.865287:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017478.865289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017478.865292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334288256:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017479.304758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017480.306726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017481.308734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017482.310797:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017483.312829:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017483.872546:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017483.872554:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017483.872557:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017483.872563:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1750 00000100:00080000:2.0:1734017483.872570:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017483.872585:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017483.872611:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334288384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017483.872631:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017483.872633:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017483.872634:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017483.872637:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1750 00000100:00080000:2.0:1734017483.872638:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017483.872640:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017483.872647:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017483.872653:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334288512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017483.872658:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017483.872660:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017483.872661:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017483.872663:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1750 00000100:00080000:2.0:1734017483.872665:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017483.872667:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017483.872670:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017483.872675:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334288640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017483.872680:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017483.872681:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017483.872682:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017483.872684:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1750 00000100:00080000:2.0:1734017483.872685:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017483.872687:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017483.872689:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017483.872694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334288768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017483.872695:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334288384:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017483.872698:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017483.872700:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017483.872701:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017483.872703:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1750 00000100:00080000:2.0:1734017483.872704:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017483.872705:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017483.872708:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017483.872712:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334288896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017483.872716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017483.872718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017483.872719:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017483.872721:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1750 00000100:00080000:2.0:1734017483.872722:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017483.872724:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017483.872726:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017483.872730:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334289024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017483.872734:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017483.872736:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017483.872736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017483.872738:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017483.872738:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1750 00000100:00080000:2.0:1734017483.872739:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017483.872741:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017483.872743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017483.872747:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334289152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017483.872750:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017483/real 1734017483] req@ffff8800a78a1f80 x1818247334288384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017483.872765:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334288384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017483.872770:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017483.872772:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017483.872775:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334288384:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017483.872779:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334288512:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017483.872788:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017483.872791:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017483/real 1734017483] req@ffff8800a78a0a80 x1818247334288512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017483.872797:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334288512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017483.872801:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017483.872802:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017483.872804:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334288512:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017483.872807:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334288640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017483.872815:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017483.872818:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017483/real 1734017483] req@ffff8800a78a2680 x1818247334288640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017483.872823:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334288640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017483.872826:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017483.872828:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017483.872830:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334288640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017483.872844:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334288768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017483.872851:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017483.872854:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017483/real 1734017483] req@ffff8800a78a0700 x1818247334288768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017483.872860:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334288768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017483.872863:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017483.872864:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017483.872866:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334288768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017483.872869:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334288896:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017483.872875:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017483.872878:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017483/real 1734017483] req@ffff8800a78a1880 x1818247334288896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017483.872883:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334288896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017483.872886:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017483.872887:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017483.872889:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334288896:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017483.872892:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334289024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017483.872900:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017483.872903:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017483/real 1734017483] req@ffff8800a78a3b80 x1818247334289024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017483.872908:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334289024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017483.872911:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017483.872912:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017483.872914:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334289024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017483.872916:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334289152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017483.872923:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017483.872926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017483/real 1734017483] req@ffff8800a78a3480 x1818247334289152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017483.872931:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334289152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017483.872934:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017483.872935:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017483.872937:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334289152:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017484.314797:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017485.316729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017486.318722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017487.320750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017488.322810:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017488.880610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017488.880621:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017488.880625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017488.880637:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1755 00000100:00080000:2.0:1734017488.880646:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017488.880654:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017488.880692:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334289280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017488.880734:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017488.880738:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017488.880740:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017488.880744:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1755 00000100:00080000:2.0:1734017488.880748:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017488.880752:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017488.880762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017488.880774:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334289408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017488.880783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017488.880786:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017488.880788:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017488.880792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1755 00000100:00080000:2.0:1734017488.880794:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017488.880798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017488.880803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017488.880811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334289536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017488.880820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017488.880823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017488.880824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017488.880828:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1755 00000100:00080000:2.0:1734017488.880830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017488.880834:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017488.880836:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334289280:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017488.880838:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017488.880847:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334289664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017488.880855:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017488.880858:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017488.880860:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017488.880863:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1755 00000100:00080000:2.0:1734017488.880865:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017488.880869:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017488.880873:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017488.880880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334289792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017488.880887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017488.880890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017488.880892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017488.880895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1755 00000100:00080000:2.0:1734017488.880898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017488.880901:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017488.880901:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017488.880906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017488.880913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334289920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017488.880921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017488.880923:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017488/real 1734017488] req@ffff8800a78a3480 x1818247334289280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017488.880924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017488.880926:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017488.880929:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1755 00000100:00080000:2.0:1734017488.880932:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017488.880935:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017488.880939:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017488.880947:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334290048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017488.880950:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334289280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017488.880960:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017488.880963:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017488.880968:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334289280:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017488.880975:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334289408:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017488.880992:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017488.880998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017488/real 1734017488] req@ffff8800a78a3b80 x1818247334289408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017488.881009:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334289408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017488.881017:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017488.881020:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017488.881023:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334289408:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017488.881028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334289536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017488.881042:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017488.881048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017488/real 1734017488] req@ffff8800a78a1880 x1818247334289536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017488.881058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334289536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017488.881065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017488.881069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017488.881072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334289536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017488.881095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334289664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017488.881109:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017488.881115:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017488/real 1734017488] req@ffff8800a78a0700 x1818247334289664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017488.881126:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334289664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017488.881133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017488.881135:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017488.881138:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334289664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017488.881143:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334289792:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017488.881156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017488.881162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017488/real 1734017488] req@ffff8800a78a2680 x1818247334289792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017488.881172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334289792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017488.881179:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017488.881181:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017488.881184:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334289792:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017488.881190:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334289920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017488.881205:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017488.881211:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017488/real 1734017488] req@ffff8800a78a0a80 x1818247334289920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017488.881221:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334289920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017488.881260:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017488.881262:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017488.881265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334289920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017488.881272:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334290048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017488.881288:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017488.881296:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017488/real 1734017488] req@ffff8800a78a1f80 x1818247334290048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017488.881306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334290048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017488.881313:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017488.881316:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017488.881319:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334290048:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017489.324738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017490.326786:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017491.328730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017492.330810:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017493.332765:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017493.888612:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017493.888623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017493.888628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017493.888639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1760 00000100:00080000:2.0:1734017493.888648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017493.888656:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017493.888695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334290176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017493.888728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017493.888732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017493.888734:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017493.888738:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1760 00000100:00080000:2.0:1734017493.888742:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017493.888745:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017493.888756:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017493.888767:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334290304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017493.888778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017493.888781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017493.888783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017493.888787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1760 00000100:00080000:2.0:1734017493.888789:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017493.888793:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017493.888798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017493.888806:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334290432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017493.888815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017493.888819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017493.888820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017493.888823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1760 00000100:00080000:2.0:1734017493.888826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017493.888829:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017493.888833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017493.888834:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334290176:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017493.888841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334290560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017493.888850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017493.888853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017493.888854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017493.888857:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1760 00000100:00080000:2.0:1734017493.888860:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017493.888864:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017493.888869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017493.888876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334290688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017493.888884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017493.888887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017493.888888:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017493.888892:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1760 00000100:00080000:2.0:1734017493.888895:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017493.888898:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017493.888900:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017493.888902:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017493.888909:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334290816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017493.888917:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017493.888920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017493.888921:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017493/real 1734017493] req@ffff8800a78a1f80 x1818247334290176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017493.888921:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017493.888925:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1760 00000100:00080000:2.0:1734017493.888927:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017493.888930:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017493.888934:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017493.888941:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334290944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017493.888949:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334290176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017493.888959:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017493.888962:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017493.888967:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334290176:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017493.888974:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334290304:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017493.888991:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017493.888998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017493/real 1734017493] req@ffff8800a78a0a80 x1818247334290304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017493.889009:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334290304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017493.889016:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017493.889019:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017493.889022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334290304:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017493.889028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334290432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017493.889041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017493.889048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017493/real 1734017493] req@ffff8800a78a2680 x1818247334290432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017493.889058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334290432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017493.889065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017493.889068:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017493.889071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334290432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017493.889101:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334290560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017493.889114:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017493.889121:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017493/real 1734017493] req@ffff8800a78a0700 x1818247334290560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017493.889131:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334290560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017493.889138:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017493.889176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017493.889180:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334290560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017493.889187:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334290688:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017493.889204:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017493.889212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017493/real 1734017493] req@ffff8800a78a1880 x1818247334290688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017493.889223:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334290688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017493.889230:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017493.889233:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017493.889236:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334290688:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017493.889241:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334290816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017493.889261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017493.889268:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017493/real 1734017493] req@ffff8800a78a3b80 x1818247334290816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017493.889278:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334290816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017493.889286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017493.889288:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017493.889291:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334290816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017493.889296:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334290944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017493.889309:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017493.889315:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017493/real 1734017493] req@ffff8800a78a3480 x1818247334290944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017493.889325:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334290944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017493.889331:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017493.889334:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017493.889337:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334290944:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017494.334792:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017495.336716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017496.338775:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017497.340759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017498.342762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017498.896546:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017498.896557:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017498.896562:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017498.896572:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1765 00000100:00080000:2.0:1734017498.896582:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017498.896589:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017498.896626:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334291072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017498.896659:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017498.896663:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017498.896665:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017498.896670:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1765 00000100:00080000:2.0:1734017498.896673:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017498.896677:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017498.896687:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017498.896698:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334291200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017498.896708:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017498.896711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017498.896713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017498.896717:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1765 00000100:00080000:2.0:1734017498.896720:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017498.896723:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017498.896729:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334291072:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017498.896729:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017498.896738:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334291328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017498.896748:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017498.896751:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017498.896752:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017498.896756:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1765 00000100:00080000:2.0:1734017498.896758:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017498.896761:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017498.896765:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017498.896773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334291456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017498.896782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017498.896784:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017498.896786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017498.896789:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1765 00000100:00080000:2.0:1734017498.896792:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017498.896796:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017498.896796:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017498.896800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017498.896807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334291584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017498.896815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017498.896817:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017498/real 1734017498] req@ffff8800a78a3480 x1818247334291072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017498.896818:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017498.896820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017498.896823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1765 00000100:00080000:2.0:1734017498.896826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017498.896830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017498.896834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017498.896840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334291712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017498.896848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017498.896851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017498.896852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017498.896854:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334291072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017498.896856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1765 00000100:00080000:2.0:1734017498.896858:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017498.896861:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017498.896865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017498.896869:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017498.896872:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734017498.896872:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334291840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017498.896878:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334291072:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017498.896887:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334291200:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017498.896909:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017498.896917:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017498/real 1734017498] req@ffff8800a78a3b80 x1818247334291200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017498.896929:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334291200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017498.896936:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017498.896939:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017498.896943:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334291200:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017498.896966:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334291328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017498.896980:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017498.896986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017498/real 1734017498] req@ffff8800a78a1880 x1818247334291328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017498.896997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334291328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017498.897005:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017498.897008:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017498.897011:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334291328:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017498.897018:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334291456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017498.897030:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017498.897036:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017498/real 1734017498] req@ffff8800a78a0700 x1818247334291456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017498.897046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334291456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017498.897053:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017498.897079:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017498.897083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334291456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017498.897090:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334291584:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017498.897105:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017498.897112:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017498/real 1734017498] req@ffff8800a78a2680 x1818247334291584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017498.897123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334291584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017498.897130:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017498.897132:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017498.897136:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334291584:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017498.897141:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334291712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017498.897158:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017498.897164:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017498/real 1734017498] req@ffff8800a78a0a80 x1818247334291712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017498.897174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334291712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017498.897181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017498.897183:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017498.897187:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334291712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017498.897193:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334291840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017498.897205:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017498.897211:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017498/real 1734017498] req@ffff8800a78a1f80 x1818247334291840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017498.897221:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334291840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017498.897228:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017498.897231:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017498.897234:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334291840:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017499.344771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017500.346742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017501.348735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017502.350772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017503.352777:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017503.904617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017503.904629:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017503.904633:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017503.904643:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1770 00000100:00080000:2.0:1734017503.904652:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017503.904660:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017503.904697:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334291968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017503.904739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017503.904743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017503.904744:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017503.904749:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1770 00000100:00080000:2.0:1734017503.904752:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017503.904757:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017503.904766:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017503.904778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334292096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017503.904788:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017503.904791:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017503.904792:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017503.904797:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1770 00000100:00080000:2.0:1734017503.904799:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017503.904803:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017503.904808:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017503.904817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334292224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017503.904825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017503.904829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017503.904830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017503.904834:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1770 00000100:00080000:2.0:1734017503.904836:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017503.904840:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017503.904843:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334291968:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017503.904844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017503.904852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334292352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017503.904861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017503.904864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017503.904866:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017503.904869:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1770 00000100:00080000:2.0:1734017503.904871:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017503.904874:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017503.904878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017503.904885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334292480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017503.904893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017503.904896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017503.904897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017503.904901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1770 00000100:00080000:2.0:1734017503.904903:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017503.904907:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017503.904908:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017503.904911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017503.904919:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334292608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017503.904927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017503.904929:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017503/real 1734017503] req@ffff8800a78a1f80 x1818247334291968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017503.904931:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017503.904933:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017503.904936:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1770 00000100:00080000:2.0:1734017503.904939:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017503.904942:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017503.904946:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017503.904953:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334292736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017503.904957:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334291968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017503.904967:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017503.904997:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017503.905002:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334291968:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017503.905010:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334292096:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017503.905030:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017503.905038:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017503/real 1734017503] req@ffff8800a78a0a80 x1818247334292096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017503.905049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334292096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017503.905057:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017503.905059:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017503.905063:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334292096:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017503.905069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334292224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017503.905083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017503.905089:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017503/real 1734017503] req@ffff8800a78a2680 x1818247334292224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017503.905099:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334292224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017503.905107:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017503.905110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017503.905113:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334292224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017503.905136:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334292352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017503.905150:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017503.905156:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017503/real 1734017503] req@ffff8800a78a0700 x1818247334292352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017503.905167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334292352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017503.905174:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017503.905176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017503.905179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334292352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017503.905184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334292480:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017503.905197:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017503.905203:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017503/real 1734017503] req@ffff8800a78a1880 x1818247334292480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017503.905213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334292480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017503.905220:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017503.905222:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017503.905225:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334292480:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017503.905231:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334292608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017503.905246:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017503.905252:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017503/real 1734017503] req@ffff8800a78a3b80 x1818247334292608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017503.905263:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334292608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017503.905270:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017503.905272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017503.905275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334292608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017503.905280:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334292736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017503.905292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017503.905298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017503/real 1734017503] req@ffff8800a78a3480 x1818247334292736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017503.905308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334292736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017503.905315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017503.905317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017503.905320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334292736:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017504.354867:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017505.356766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017506.358733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017507.360792:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017508.362763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017508.912597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017508.912608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017508.912612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017508.912623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1775 00000100:00080000:2.0:1734017508.912632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017508.912640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017508.912679:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334292864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017508.912713:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017508.912717:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017508.912719:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017508.912723:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1775 00000100:00080000:2.0:1734017508.912727:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017508.912730:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017508.912741:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017508.912752:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334292992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017508.912763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017508.912766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017508.912768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017508.912771:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1775 00000100:00080000:2.0:1734017508.912774:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017508.912777:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017508.912782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017508.912791:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334293120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017508.912800:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017508.912803:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017508.912804:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017508.912807:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1775 00000100:00080000:2.0:1734017508.912810:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017508.912813:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017508.912817:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017508.912821:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334292864:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017508.912825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334293248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017508.912835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017508.912837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017508.912839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017508.912842:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1775 00000100:00080000:2.0:1734017508.912845:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017508.912848:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017508.912852:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017508.912860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334293376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017508.912868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017508.912871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017508.912872:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017508.912876:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1775 00000100:00080000:2.0:1734017508.912879:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017508.912882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017508.912918:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017508.912925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017508.912934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334293504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017508.912942:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017508/real 1734017508] req@ffff8800a78a3480 x1818247334292864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017508.912943:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017508.912946:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017508.912947:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017508.912951:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1775 00000100:00080000:2.0:1734017508.912954:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017508.912957:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017508.912961:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017508.912969:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334292864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017508.912969:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334293632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017508.912979:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017508.912982:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017508.912987:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334292864:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017508.912995:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334292992:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017508.913014:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017508.913021:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017508/real 1734017508] req@ffff8800a78a3b80 x1818247334292992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017508.913032:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334292992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017508.913039:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017508.913042:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017508.913045:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334292992:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017508.913051:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334293120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017508.913066:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017508.913072:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017508/real 1734017508] req@ffff8800a78a1880 x1818247334293120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017508.913082:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334293120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017508.913089:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017508.913093:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017508.913096:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334293120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017508.913120:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334293248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017508.913135:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017508.913141:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017508/real 1734017508] req@ffff8800a78a0700 x1818247334293248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017508.913151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334293248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017508.913158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017508.913160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017508.913163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334293248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017508.913169:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334293376:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017508.913181:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017508.913187:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017508/real 1734017508] req@ffff8800a78a2680 x1818247334293376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017508.913197:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334293376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017508.913204:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017508.913206:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017508.913209:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334293376:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017508.913215:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334293504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017508.913230:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017508.913237:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017508/real 1734017508] req@ffff8800a78a0a80 x1818247334293504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017508.913247:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334293504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017508.913254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017508.913256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017508.913259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334293504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017508.913264:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334293632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017508.913277:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017508.913283:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017508/real 1734017508] req@ffff8800a78a1f80 x1818247334293632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017508.913292:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334293632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017508.913299:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017508.913301:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017508.913304:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334293632:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017509.364741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017510.366784:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017511.368728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017512.370779:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017513.372760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017513.920600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017513.920611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017513.920616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017513.920627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1780 00000100:00080000:2.0:1734017513.920635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017513.920643:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017513.920681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334293760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017513.920716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017513.920720:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017513.920722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017513.920727:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1780 00000100:00080000:2.0:1734017513.920730:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017513.920734:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017513.920744:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017513.920755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334293888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017513.920765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017513.920768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017513.920770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017513.920774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1780 00000100:00080000:2.0:1734017513.920777:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017513.920780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017513.920786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017513.920795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334294016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017513.920836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017513.920839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017513.920841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017513.920845:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1780 00000100:00080000:2.0:1734017513.920848:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017513.920851:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017513.920855:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334293760:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017513.920856:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017513.920865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334294144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017513.920874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017513.920877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017513.920879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017513.920882:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1780 00000100:00080000:2.0:1734017513.920884:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017513.920887:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017513.920892:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017513.920899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334294272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017513.920907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017513.920910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017513.920912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017513.920915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1780 00000100:00080000:2.0:1734017513.920918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017513.920922:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017513.920922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017513.920926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017513.920933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334294400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017513.920941:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017513.920943:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017513/real 1734017513] req@ffff8800a78a1f80 x1818247334293760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017513.920944:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017513.920945:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017513.920948:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1780 00000100:00080000:2.0:1734017513.920951:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017513.920954:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017513.920958:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017513.920965:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334294528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017513.920970:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334293760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017513.920980:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017513.920984:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017513.920988:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334293760:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017513.920995:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334293888:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017513.921012:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017513.921019:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017513/real 1734017513] req@ffff8800a78a0a80 x1818247334293888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017513.921030:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334293888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017513.921037:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017513.921040:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017513.921043:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334293888:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017513.921049:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334294016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017513.921062:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017513.921069:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017513/real 1734017513] req@ffff8800a78a2680 x1818247334294016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017513.921078:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334294016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017513.921085:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017513.921089:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017513.921092:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334294016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017513.921117:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334294144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017513.921130:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017513.921136:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017513/real 1734017513] req@ffff8800a78a0700 x1818247334294144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017513.921147:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334294144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017513.921154:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017513.921156:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017513.921159:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334294144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017513.921165:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334294272:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017513.921178:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017513.921184:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017513/real 1734017513] req@ffff8800a78a1880 x1818247334294272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017513.921194:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334294272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017513.921201:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017513.921203:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017513.921206:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334294272:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017513.921211:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334294400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017513.921226:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017513.921232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017513/real 1734017513] req@ffff8800a78a3b80 x1818247334294400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017513.921243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334294400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017513.921250:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017513.921252:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017513.921255:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334294400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017513.921260:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334294528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017513.921273:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017513.921279:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017513/real 1734017513] req@ffff8800a78a3480 x1818247334294528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017513.921288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334294528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017513.921295:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017513.921297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017513.921300:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334294528:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017514.374833:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017515.376718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017516.378740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017517.380802:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017518.382819:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017518.928626:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017518.928637:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017518.928642:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017518.928655:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1785 00000100:00080000:2.0:1734017518.928667:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017518.928676:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017518.928767:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334294656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017518.928811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017518.928816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017518.928818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017518.928823:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1785 00000100:00080000:2.0:1734017518.928827:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017518.928831:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017518.928841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017518.928855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334294784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017518.928865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017518.928868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017518.928870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017518.928874:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1785 00000100:00080000:2.0:1734017518.928877:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017518.928880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017518.928885:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017518.928894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334294912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017518.928903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017518.928906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017518.928908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017518.928912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1785 00000100:00080000:2.0:1734017518.928914:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017518.928918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017518.928922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017518.928932:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334295040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017518.928941:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017518.928944:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017518.928946:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017518.928950:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334294656:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017518.928950:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1785 00000100:00080000:2.0:1734017518.928952:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017518.928956:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017518.928960:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017518.928969:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334295168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017518.928977:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017518.928980:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017518.928982:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017518.928985:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1785 00000100:00080000:2.0:1734017518.928988:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017518.928991:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017518.928995:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017518.929003:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334295296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017518.929011:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017518.929015:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017518.929017:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017518.929021:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1785 00000100:00080000:2.0:1734017518.929025:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017518.929027:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017518.929028:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017518.929032:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017518.929041:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334295424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017518.929052:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017518/real 1734017518] req@ffff8800a78a3480 x1818247334294656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017518.929080:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334294656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017518.929092:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017518.929095:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017518.929100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334294656:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017518.929108:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334294784:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017518.929126:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017518.929133:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017518/real 1734017518] req@ffff8800a78a3b80 x1818247334294784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017518.929144:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334294784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017518.929152:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017518.929155:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017518.929159:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334294784:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017518.929165:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334294912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017518.929179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017518.929185:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017518/real 1734017518] req@ffff8800a78a1880 x1818247334294912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017518.929195:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334294912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017518.929203:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017518.929206:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017518.929209:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334294912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017518.929236:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334295040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017518.929250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017518.929256:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017518/real 1734017518] req@ffff8800a78a0700 x1818247334295040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017518.929266:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334295040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017518.929274:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017518.929276:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017518.929279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334295040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017518.929286:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334295168:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017518.929299:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017518.929305:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017518/real 1734017518] req@ffff8800a78a2680 x1818247334295168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017518.929315:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334295168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017518.929322:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017518.929324:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017518.929327:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334295168:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017518.929334:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334295296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017518.929349:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017518.929356:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017518/real 1734017518] req@ffff8800a78a0a80 x1818247334295296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017518.929366:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334295296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017518.929373:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017518.929375:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017518.929378:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334295296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017518.929384:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334295424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017518.929397:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017518.929403:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017518/real 1734017518] req@ffff8800a78a1f80 x1818247334295424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017518.929413:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334295424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017518.929420:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017518.929422:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017518.929425:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334295424:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017519.384798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017520.386733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017521.388764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017522.390769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017523.392772:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017523.936605:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017523.936616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017523.936621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017523.936667:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1790 00000100:00080000:2.0:1734017523.936677:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017523.936684:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017523.936723:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334295552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017523.936757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017523.936761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017523.936764:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017523.936768:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1790 00000100:00080000:2.0:1734017523.936771:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017523.936775:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017523.936785:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017523.936797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334295680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017523.936808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017523.936811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017523.936813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017523.936817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1790 00000100:00080000:2.0:1734017523.936819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017523.936823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017523.936827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017523.936836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334295808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017523.936844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017523.936848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017523.936849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017523.936853:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1790 00000100:00080000:2.0:1734017523.936855:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017523.936858:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017523.936862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017523.936871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334295936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017523.936875:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334295552:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017523.936881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017523.936884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017523.936885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017523.936888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1790 00000100:00080000:2.0:1734017523.936891:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017523.936894:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017523.936898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017523.936905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334296064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017523.936913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017523.936916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017523.936918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017523.936921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1790 00000100:00080000:2.0:1734017523.936924:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017523.936927:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017523.936930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017523.936938:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334296192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017523.936940:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017523.936945:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017523.936948:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017523.936950:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017523.936953:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1790 00000100:00080000:2.0:1734017523.936957:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017523.936961:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017523.936962:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017523/real 1734017523] req@ffff8800a78a1f80 x1818247334295552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734017523.936964:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017523.936972:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334296320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017523.936990:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334295552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017523.937000:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017523.937003:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017523.937008:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334295552:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017523.937015:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334295680:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017523.937032:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017523.937039:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017523/real 1734017523] req@ffff8800a78a0a80 x1818247334295680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017523.937050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334295680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017523.937058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017523.937060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017523.937064:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334295680:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017523.937070:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334295808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017523.937083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017523.937089:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017523/real 1734017523] req@ffff8800a78a2680 x1818247334295808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017523.937099:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334295808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017523.937106:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017523.937110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017523.937113:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334295808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017523.937138:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334295936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017523.937151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017523.937158:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017523/real 1734017523] req@ffff8800a78a0700 x1818247334295936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017523.937168:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334295936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017523.937175:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017523.937178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017523.937181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334295936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017523.937186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334296064:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017523.937199:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017523.937204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017523/real 1734017523] req@ffff8800a78a1880 x1818247334296064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017523.937214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334296064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017523.937221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017523.937223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017523.937226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334296064:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017523.937232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334296192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017523.937247:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017523.937253:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017523/real 1734017523] req@ffff8800a78a3b80 x1818247334296192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017523.937263:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334296192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017523.937271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017523.937273:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017523.937276:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334296192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017523.937281:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334296320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017523.937293:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017523.937299:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017523/real 1734017523] req@ffff8800a78a3480 x1818247334296320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017523.937309:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334296320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017523.937316:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017523.937318:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017523.937321:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334296320:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017524.394754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017525.396825:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017526.398715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017527.400714:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017528.402749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017528.944647:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017528.944656:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017528.944660:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017528.944671:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1795 00000100:00080000:2.0:1734017528.944680:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017528.944688:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017528.944726:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334296448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017528.944761:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017528.944766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017528.944767:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017528.944772:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1795 00000100:00080000:2.0:1734017528.944775:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017528.944779:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017528.944789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017528.944801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334296576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017528.944811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017528.944814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017528.944816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017528.944820:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1795 00000100:00080000:2.0:1734017528.944823:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017528.944826:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017528.944832:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017528.944841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334296704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017528.944850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017528.944853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017528.944854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017528.944858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1795 00000100:00080000:2.0:1734017528.944860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017528.944863:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017528.944867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017528.944874:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334296448:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017528.944875:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334296832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017528.944884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017528.944887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017528.944889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017528.944892:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1795 00000100:00080000:2.0:1734017528.944894:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017528.944897:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017528.944901:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017528.944909:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334296960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017528.944917:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017528.944920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017528.944922:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017528.944926:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1795 00000100:00080000:2.0:1734017528.944929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017528.944932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017528.944936:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017528.944940:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017528.944944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334297088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017528.944952:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017528.944955:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017528.944956:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017528.944959:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1795 00000100:00100000:1.0:1734017528.944962:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017528/real 1734017528] req@ffff8800a78a3480 x1818247334296448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017528.944962:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017528.944965:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017528.944969:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017528.944976:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334297216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017528.944989:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334296448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017528.944999:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017528.945002:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017528.945007:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334296448:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017528.945014:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334296576:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017528.945031:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017528.945038:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017528/real 1734017528] req@ffff8800a78a3b80 x1818247334296576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017528.945049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334296576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017528.945056:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017528.945059:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017528.945062:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334296576:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017528.945068:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334296704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017528.945081:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017528.945087:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017528/real 1734017528] req@ffff8800a78a1880 x1818247334296704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017528.945097:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334296704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017528.945104:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017528.945107:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017528.945110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334296704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017528.945134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334296832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017528.945147:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017528.945154:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017528/real 1734017528] req@ffff8800a78a0700 x1818247334296832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017528.945164:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334296832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017528.945172:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017528.945174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017528.945177:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334296832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017528.945182:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334296960:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017528.945195:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017528.945201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017528/real 1734017528] req@ffff8800a78a2680 x1818247334296960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017528.945210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334296960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017528.945218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017528.945220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017528.945223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334296960:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017528.945228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334297088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017528.945244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017528.945250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017528/real 1734017528] req@ffff8800a78a0a80 x1818247334297088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017528.945261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334297088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017528.945268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017528.945270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017528.945273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334297088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017528.945279:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334297216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017528.945291:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017528.945297:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017528/real 1734017528] req@ffff8800a78a1f80 x1818247334297216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017528.945307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334297216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017528.945314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017528.945317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017528.945320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334297216:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017529.404703:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017530.406732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017531.408707:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017532.410722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017533.412787:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017533.952599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017533.952608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017533.952612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017533.952622:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1800 00000100:00080000:2.0:1734017533.952631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017533.952639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017533.952677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334297344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017533.952719:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017533.952723:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017533.952725:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017533.952730:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1800 00000100:00080000:2.0:1734017533.952733:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017533.952737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017533.952747:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017533.952759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334297472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017533.952768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017533.952771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017533.952773:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017533.952777:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1800 00000100:00080000:2.0:1734017533.952779:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017533.952783:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017533.952788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017533.952796:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334297600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017533.952805:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017533.952808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017533.952810:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017533.952813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1800 00000100:00080000:2.0:1734017533.952815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017533.952819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017533.952823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017533.952831:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334297344:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017533.952832:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334297728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017533.952841:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017533.952844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017533.952845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017533.952848:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1800 00000100:00080000:2.0:1734017533.952851:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017533.952854:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017533.952858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017533.952865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334297856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017533.952873:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017533.952876:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017533.952878:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017533.952881:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1800 00000100:00080000:2.0:1734017533.952893:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017533.952896:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017533.952900:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017533.952908:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334297984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017533.952917:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017533.952920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017533.952922:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017533.952923:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334297472:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017533.952925:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1800 00000100:00080000:2.0:1734017533.952928:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017533.952931:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017533.952935:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017533.952942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334298112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017533.952943:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334297600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017533.952974:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334297728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017533.952991:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334297856:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017533.953005:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334297984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017533.953020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334298112:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017534.414747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017535.416761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017536.418710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017536.958762:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017536.958811:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017536.958829:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017536.958867:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017536.958874:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017536.958881:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017536.958887:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017536.958894:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017536.958900:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734017536.958933:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017533/real 1734017536] req@ffff8800a78a0700 x1818247334297728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017536.958980:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334297728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017536.958998:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017536.959002:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017536.959008:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334297728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017536.959016:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017533/real 1734017536] req@ffff8800a78a1880 x1818247334297856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017536.959030:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334297856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017536.959039:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017536.959043:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017536.959049:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334297856:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017536.959060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017533/real 1734017536] req@ffff8800a78a3b80 x1818247334297984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017536.959076:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334297984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017536.959088:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017536.959092:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017536.959097:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334297984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017536.959106:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017533/real 1734017536] req@ffff8800a78a3480 x1818247334298112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017536.959123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334298112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017536.959135:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017536.959139:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017536.959144:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334298112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017536.959153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017533/real 1734017536] req@ffff8800a78a1f80 x1818247334297344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017536.959171:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334297344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017536.959184:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017536.959187:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017536.959193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334297344:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017536.959203:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017533/real 1734017536] req@ffff8800a78a0a80 x1818247334297472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017536.959222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334297472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017536.959245:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017536.959249:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017536.959254:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334297472:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017536.959265:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017533/real 1734017536] req@ffff8800a78a2680 x1818247334297600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017536.959283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334297600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017536.959294:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017536.959298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017536.959303:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334297600:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017537.420742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017538.422758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017538.960541:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017538.960554:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017538.960560:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017538.960574:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1805 00000100:00080000:2.0:1734017538.960587:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017538.960596:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017538.960645:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334298240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017538.960689:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017538.960694:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017538.960696:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017538.960700:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1805 00000100:00080000:2.0:1734017538.960704:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017538.960707:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017538.960718:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017538.960731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334298368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017538.960742:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017538.960745:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017538.960747:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017538.960751:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1805 00000100:00080000:2.0:1734017538.960755:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017538.960761:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017538.960769:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017538.960786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334298496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017538.960802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017538.960807:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017538.960809:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017538.960815:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1805 00000100:00080000:2.0:1734017538.960819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017538.960821:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334298240:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017538.960825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017538.960832:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017538.960848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334298624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017538.960864:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017538.960869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017538.960871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017538.960878:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1805 00000100:00080000:2.0:1734017538.960884:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017538.960889:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017538.960897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734017538.960908:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017538.960925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334298752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017538.960934:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017538/real 1734017538] req@ffff8800a78a2680 x1818247334298240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017538.960937:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017538.960940:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017538.960942:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017538.960946:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1805 00000100:00080000:2.0:1734017538.960950:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017538.960955:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017538.960956:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334298240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734017538.960961:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017538.960967:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017538.960971:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734017538.960971:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334298880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017538.960976:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334298240:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017538.960980:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017538.960984:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017538.960986:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334298368:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017538.960986:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017538.960989:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1805 00000100:00080000:2.0:1734017538.960992:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017538.960995:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017538.961000:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017538.961008:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017538.961009:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334299008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017538.961018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017538/real 1734017538] req@ffff8800a78a0a80 x1818247334298368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017538.961031:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334298368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017538.961039:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017538.961042:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017538.961045:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334298368:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017538.961071:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334298496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017538.961086:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017538.961093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017538/real 1734017538] req@ffff8800a78a1f80 x1818247334298496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017538.961103:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334298496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017538.961111:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017538.961113:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017538.961117:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334298496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017538.961123:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334298624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017538.961136:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017538.961142:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017538/real 1734017538] req@ffff8800a78a3480 x1818247334298624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017538.961152:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334298624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017538.961167:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017538.961170:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017538.961173:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334298624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017538.961179:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334298752:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017538.961192:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017538.961198:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017538/real 1734017538] req@ffff8800a78a3b80 x1818247334298752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017538.961208:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334298752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017538.961216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017538.961219:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017538.961222:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334298752:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017538.961229:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334298880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017538.961241:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017538.961247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017538/real 1734017538] req@ffff8800a78a1880 x1818247334298880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017538.961257:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334298880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017538.961264:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017538.961266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017538.961269:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334298880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017538.961276:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334299008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017538.961290:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017538.961296:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017538/real 1734017538] req@ffff8800a78a0700 x1818247334299008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017538.961306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334299008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017538.961313:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017538.961315:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017538.961318:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334299008:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017539.424780:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017540.426733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017541.428758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017542.430788:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017543.432775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017543.968582:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017543.968591:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017543.968596:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017543.968606:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1810 00000100:00080000:2.0:1734017543.968615:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017543.968623:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017543.968662:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334299136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017543.968697:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017543.968701:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017543.968703:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017543.968708:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1810 00000100:00080000:2.0:1734017543.968711:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017543.968715:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017543.968725:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017543.968737:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334299264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017543.968755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017543.968758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017543.968760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017543.968764:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1810 00000100:00080000:2.0:1734017543.968767:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017543.968770:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017543.968776:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017543.968785:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334299392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017543.968794:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017543.968797:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017543.968799:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017543.968802:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1810 00000100:00080000:2.0:1734017543.968805:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017543.968808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017543.968812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017543.968814:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334299136:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017543.968820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334299520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017543.968829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017543.968832:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017543.968833:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017543.968836:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1810 00000100:00080000:2.0:1734017543.968839:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017543.968842:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017543.968846:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017543.968853:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334299648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017543.968861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017543.968864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017543.968866:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017543.968869:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1810 00000100:00080000:2.0:1734017543.968873:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017543.968876:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017543.968880:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017543.968881:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017543.968887:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334299776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017543.968895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017543.968898:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017543.968900:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017543.968902:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017543/real 1734017543] req@ffff8800a78a0700 x1818247334299136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017543.968903:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1810 00000100:00080000:2.0:1734017543.968906:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017543.968909:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017543.968913:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017543.968920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334299904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017543.968922:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334299136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017543.968933:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017543.968936:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017543.968941:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334299136:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017543.968948:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334299264:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017543.968964:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017543.968971:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017543/real 1734017543] req@ffff8800a78a1880 x1818247334299264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017543.968991:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334299264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017543.968998:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017543.969001:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017543.969004:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334299264:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017543.969010:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334299392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017543.969023:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017543.969029:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017543/real 1734017543] req@ffff8800a78a3b80 x1818247334299392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017543.969039:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334299392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017543.969046:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017543.969048:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017543.969051:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334299392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017543.969076:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334299520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017543.969089:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017543.969095:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017543/real 1734017543] req@ffff8800a78a3480 x1818247334299520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017543.969106:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334299520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017543.969113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017543.969116:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017543.969119:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334299520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017543.969125:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334299648:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017543.969138:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017543.969143:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017543/real 1734017543] req@ffff8800a78a1f80 x1818247334299648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017543.969154:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334299648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017543.969161:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017543.969164:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017543.969167:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334299648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017543.969172:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334299776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017543.969184:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017543.969190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017543/real 1734017543] req@ffff8800a78a0a80 x1818247334299776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017543.969200:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334299776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017543.969207:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017543.969209:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017543.969212:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334299776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017543.969217:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334299904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017543.969232:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017543.969238:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017543/real 1734017543] req@ffff8800a78a2680 x1818247334299904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017543.969248:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334299904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017543.969255:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017543.969257:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017543.969260:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334299904:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017544.434713:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017545.436750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017546.438703:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017547.440770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017548.442734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017548.976603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017548.976611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017548.976616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017548.976626:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1815 00000100:00080000:2.0:1734017548.976635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017548.976643:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017548.976680:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334300032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017548.976723:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017548.976727:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017548.976729:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017548.976734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1815 00000100:00080000:2.0:1734017548.976737:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017548.976741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017548.976751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017548.976763:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334300160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017548.976772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017548.976776:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017548.976777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017548.976781:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1815 00000100:00080000:2.0:1734017548.976784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017548.976787:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017548.976792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017548.976801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334300288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017548.976809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017548.976813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017548.976814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017548.976817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1815 00000100:00080000:2.0:1734017548.976820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017548.976824:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017548.976828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017548.976829:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334300032:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017548.976836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334300416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017548.976845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017548.976848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017548.976849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017548.976852:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1815 00000100:00080000:2.0:1734017548.976855:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017548.976858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017548.976862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017548.976869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334300544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017548.976877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017548.976880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017548.976881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017548.976884:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1815 00000100:00080000:2.0:1734017548.976887:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017548.976891:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017548.976894:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017548.976894:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017548.976902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334300672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017548.976910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017548.976914:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017548.976916:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017548/real 1734017548] req@ffff8800a78a2680 x1818247334300032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017548.976916:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017548.976919:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1815 00000100:00080000:2.0:1734017548.976923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017548.976926:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017548.976930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017548.976937:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334300032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017548.976937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334300800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017548.976947:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017548.976950:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017548.976955:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334300032:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017548.976963:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334300160:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017548.976979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017548.976986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017548/real 1734017548] req@ffff8800a78a0a80 x1818247334300160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017548.976997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334300160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017548.977006:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017548.977009:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017548.977012:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334300160:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017548.977018:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334300288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017548.977031:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017548.977037:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017548/real 1734017548] req@ffff8800a78a1f80 x1818247334300288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017548.977047:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334300288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017548.977055:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017548.977057:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017548.977060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334300288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017548.977091:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334300416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017548.977104:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017548.977110:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017548/real 1734017548] req@ffff8800a78a3480 x1818247334300416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017548.977121:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334300416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017548.977128:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017548.977130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017548.977133:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334300416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017548.977144:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334300544:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017548.977157:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017548.977163:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017548/real 1734017548] req@ffff8800a78a3b80 x1818247334300544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017548.977173:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334300544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017548.977180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017548.977182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017548.977185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334300544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017548.977190:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334300672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017548.977232:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017548.977241:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017548/real 1734017548] req@ffff8800a78a1880 x1818247334300672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017548.977251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334300672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017548.977258:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017548.977260:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017548.977263:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334300672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017548.977269:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334300800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017548.977284:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017548.977290:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017548/real 1734017548] req@ffff8800a78a0700 x1818247334300800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017548.977302:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334300800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017548.977309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017548.977311:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017548.977314:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334300800:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017549.444767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017550.446717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017551.448728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017552.450717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017553.452765:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017553.984610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017553.984619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017553.984623:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017553.984634:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1820 00000100:00080000:2.0:1734017553.984643:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017553.984650:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017553.984690:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334300928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017553.984726:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017553.984730:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017553.984733:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017553.984737:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1820 00000100:00080000:2.0:1734017553.984741:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017553.984745:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017553.984755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017553.984767:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334301056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017553.984778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017553.984781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017553.984783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017553.984787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1820 00000100:00080000:2.0:1734017553.984790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017553.984793:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017553.984798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017553.984807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334301184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017553.984816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017553.984819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017553.984821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017553.984824:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1820 00000100:00080000:2.0:1734017553.984826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017553.984829:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017553.984834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017553.984843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334301312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017553.984853:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334300928:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017553.984853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017553.984856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017553.984857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017553.984861:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1820 00000100:00080000:2.0:1734017553.984864:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017553.984867:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017553.984871:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017553.984879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334301440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017553.984887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017553.984890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017553.984892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017553.984895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1820 00000100:00080000:2.0:1734017553.984897:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017553.984900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017553.984904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017553.984911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334301568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017553.984919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017553.984922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017553.984923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017553.984926:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1820 00000100:00080000:2.0:1734017553.984939:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017553.984942:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017553.984947:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017553.984947:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017553.984954:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334301696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017553.984975:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017553/real 1734017553] req@ffff8800a78a0700 x1818247334300928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017553.984997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334300928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017553.985010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017553.985014:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017553.985019:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334300928:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017553.985027:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334301056:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017553.985046:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017553.985053:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017553/real 1734017553] req@ffff8800a78a1880 x1818247334301056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017553.985064:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334301056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017553.985072:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017553.985076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017553.985080:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334301056:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017553.985087:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334301184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017553.985102:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017553.985108:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017553/real 1734017553] req@ffff8800a78a3b80 x1818247334301184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017553.985235:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334301184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017553.985244:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017553.985247:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017553.985250:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334301184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017553.985283:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334301312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017553.985303:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017553.985311:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017553/real 1734017553] req@ffff8800a78a3480 x1818247334301312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017553.985323:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334301312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017553.985331:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017553.985334:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017553.985337:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334301312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017553.985343:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334301440:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017553.985361:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017553.985367:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017553/real 1734017553] req@ffff8800a78a1f80 x1818247334301440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017553.985377:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334301440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017553.985385:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017553.985388:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017553.985391:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334301440:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017553.985397:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334301568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017553.985409:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017553.985415:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017553/real 1734017553] req@ffff8800a78a0a80 x1818247334301568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017553.985424:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334301568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017553.985432:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017553.985434:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017553.985437:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334301568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017553.985443:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334301696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017553.985456:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017553.985462:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017553/real 1734017553] req@ffff8800a78a2680 x1818247334301696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017553.985516:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334301696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017553.985525:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017553.985527:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017553.985530:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334301696:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017554.454741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017555.456769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017556.458731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017557.460764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017558.462738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017558.992600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017558.992608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017558.992613:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017558.992624:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1825 00000100:00080000:2.0:1734017558.992633:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017558.992641:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017558.992678:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334301824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017558.992714:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017558.992718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017558.992720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017558.992724:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1825 00000100:00080000:2.0:1734017558.992727:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017558.992731:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017558.992741:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017558.992753:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334301952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017558.992763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017558.992766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017558.992767:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017558.992772:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1825 00000100:00080000:2.0:1734017558.992774:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017558.992778:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017558.992783:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017558.992792:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334302080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017558.992801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017558.992804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017558.992807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017558.992810:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1825 00000100:00080000:2.0:1734017558.992812:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017558.992815:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017558.992819:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017558.992827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334302208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017558.992836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017558.992838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017558.992840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017558.992843:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1825 00000100:00080000:2.0:1734017558.992845:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017558.992849:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017558.992853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017558.992861:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334302336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017558.992870:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334301824:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017558.992871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017558.992874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017558.992877:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017558.992880:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1825 00000100:00080000:2.0:1734017558.992883:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017558.992886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017558.992891:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017558.992898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334302464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017558.992906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017558.992909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017558.992910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017558.992914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1825 00000100:00080000:2.0:1734017558.992916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017558.992919:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017558.992923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017558.992930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334302592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017558.992970:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017558.992998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017558/real 1734017558] req@ffff8800a78a2680 x1818247334301824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017558.993021:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334301824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017558.993065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017558.993070:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017558.993075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334301824:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017558.993084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334301952:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017558.993107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017558.993114:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017558/real 1734017558] req@ffff8800a78a0a80 x1818247334301952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017558.993126:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334301952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017558.993135:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017558.993139:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017558.993143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334301952:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017558.993150:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334302080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017558.993166:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017558.993172:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017558/real 1734017558] req@ffff8800a78a1f80 x1818247334302080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017558.993182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334302080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017558.993191:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017558.993193:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017558.993197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334302080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017558.993203:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334302208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017558.993217:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017558.993223:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017558/real 1734017558] req@ffff8800a78a3480 x1818247334302208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017558.993233:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334302208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017558.993240:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017558.993243:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017558.993246:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334302208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017558.993277:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334302336:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017558.993295:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017558.993302:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017558/real 1734017558] req@ffff8800a78a3b80 x1818247334302336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017558.993313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334302336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017558.993320:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017558.993323:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017558.993326:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334302336:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017558.993332:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334302464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017558.993344:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017558.993351:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017558/real 1734017558] req@ffff8800a78a1880 x1818247334302464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017558.993361:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334302464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017558.993368:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017558.993370:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017558.993374:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334302464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017558.993380:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334302592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017558.993392:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017558.993398:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017558/real 1734017558] req@ffff8800a78a0700 x1818247334302592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017558.993416:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334302592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017558.993423:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017558.993426:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017558.993429:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334302592:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017559.464764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017560.466687:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017561.468754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017562.470773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017563.472738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017564.000615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017564.000624:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017564.000629:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017564.000640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1830 00000100:00080000:2.0:1734017564.000648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017564.000656:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017564.000694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334302720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017564.000738:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017564.000742:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017564.000744:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017564.000748:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1830 00000100:00080000:2.0:1734017564.000751:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017564.000755:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017564.000766:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017564.000777:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334302848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017564.000787:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017564.000790:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017564.000792:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017564.000796:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1830 00000100:00080000:2.0:1734017564.000798:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017564.000802:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017564.000807:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017564.000816:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334302976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017564.000824:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017564.000827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017564.000829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017564.000832:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1830 00000100:00080000:2.0:1734017564.000834:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017564.000839:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017564.000843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017564.000851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334303104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017564.000860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017564.000874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017564.000876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017564.000880:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1830 00000100:00080000:2.0:1734017564.000882:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017564.000885:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017564.000890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017564.000898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334303232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017564.000908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017564.000911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017564.000913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017564.000914:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334302720:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017564.000917:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1830 00000100:00080000:2.0:1734017564.000919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017564.000922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017564.000927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017564.000941:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334303360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017564.000979:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017564.000982:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017564.000984:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017564.000988:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1830 00000100:00080000:2.0:1734017564.000992:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017564.000995:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017564.000999:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017564.001007:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334303488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017564.001052:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017564.001082:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017564/real 1734017564] req@ffff8800a78a0700 x1818247334302720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017564.001105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334302720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017564.001118:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017564.001122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017564.001127:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334302720:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017564.001136:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334302848:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017564.001155:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017564.001162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017564/real 1734017564] req@ffff8800a78a1880 x1818247334302848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017564.001174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334302848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017564.001182:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017564.001186:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017564.001190:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334302848:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017564.001197:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334302976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017564.001212:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017564.001218:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017564/real 1734017564] req@ffff8800a78a3b80 x1818247334302976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017564.001228:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334302976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017564.001237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017564.001239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017564.001243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334302976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017564.001249:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334303104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017564.001263:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017564.001269:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017564/real 1734017564] req@ffff8800a78a3480 x1818247334303104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017564.001279:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334303104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017564.001287:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017564.001290:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017564.001293:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334303104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017564.001326:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334303232:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017564.001343:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017564.001350:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017564/real 1734017564] req@ffff8800a78a1f80 x1818247334303232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017564.001361:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334303232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017564.001369:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017564.001371:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017564.001374:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334303232:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017564.001380:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334303360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017564.001393:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017564.001399:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017564/real 1734017564] req@ffff8800a78a0a80 x1818247334303360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017564.001408:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334303360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017564.001416:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017564.001418:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017564.001421:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334303360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017564.001426:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334303488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017564.001440:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017564.001446:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017564/real 1734017564] req@ffff8800a78a2680 x1818247334303488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017564.001462:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334303488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017564.001600:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017564.001603:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017564.001607:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334303488:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017564.474765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017565.476738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017566.478774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017567.480743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017568.482749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017569.008605:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017569.008618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017569.008623:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017569.008636:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1835 00000100:00080000:2.0:1734017569.008646:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017569.008656:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017569.008705:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334303616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017569.008742:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017569.008746:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017569.008748:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017569.008753:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1835 00000100:00080000:2.0:1734017569.008756:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017569.008760:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017569.008770:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017569.008783:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334303744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017569.008794:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017569.008798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017569.008799:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017569.008803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1835 00000100:00080000:2.0:1734017569.008806:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017569.008809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017569.008814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017569.008823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334303872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017569.008833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017569.008836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017569.008837:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017569.008841:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1835 00000100:00080000:2.0:1734017569.008843:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017569.008847:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017569.008851:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017569.008854:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334303616:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017569.008885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334304000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017569.008896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017569.008899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017569.008901:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017569.008906:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1835 00000100:00080000:2.0:1734017569.008909:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017569.008912:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017569.008918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017569.008926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334304128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017569.008935:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017569.008938:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017569.008940:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017569.008944:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1835 00000100:00080000:2.0:1734017569.008947:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017569.008950:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017569.008954:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017569.008962:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334304256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017569.008970:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017569.008973:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734017569.008974:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017569.008975:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017569.008979:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1835 00000100:00080000:2.0:1734017569.008981:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017569.008984:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017569.008988:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017569.008996:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334304384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017569.009000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017569/real 1734017569] req@ffff8800a78a2680 x1818247334303616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017569.009022:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334303616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017569.009034:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017569.009038:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017569.009043:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334303616:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017569.009051:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334303744:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017569.009070:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017569.009077:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017569/real 1734017569] req@ffff8800a78a0a80 x1818247334303744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017569.009088:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334303744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017569.009096:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017569.009100:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017569.009103:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334303744:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017569.009109:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334303872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017569.009124:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017569.009130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017569/real 1734017569] req@ffff8800a78a1f80 x1818247334303872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017569.009140:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334303872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017569.009148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017569.009151:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017569.009154:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334303872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017569.009185:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334304000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017569.009198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017569.009204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017569/real 1734017569] req@ffff8800a78a3480 x1818247334304000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017569.009215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334304000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017569.009223:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017569.009226:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017569.009229:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334304000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017569.009234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334304128:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017569.009250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017569.009257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017569/real 1734017569] req@ffff8800a78a3b80 x1818247334304128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017569.009267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334304128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017569.009275:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017569.009277:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017569.009280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334304128:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017569.009285:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334304256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017569.009297:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017569.009303:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017569/real 1734017569] req@ffff8800a78a1880 x1818247334304256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017569.009313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334304256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017569.009320:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017569.009322:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017569.009325:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334304256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017569.009330:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334304384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017569.009343:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017569.009349:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017569/real 1734017569] req@ffff8800a78a0700 x1818247334304384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017569.009366:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334304384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017569.009374:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017569.009376:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017569.009379:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334304384:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017569.484752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017570.486758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017571.488657:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017572.491061:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017573.492758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017574.016602:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017574.016611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017574.016615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017574.016626:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1840 00000100:00080000:2.0:1734017574.016635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017574.016642:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017574.016680:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334304512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017574.016715:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017574.016719:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017574.016721:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017574.016725:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1840 00000100:00080000:2.0:1734017574.016729:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017574.016732:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017574.016742:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017574.016754:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334304640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017574.016764:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017574.016767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017574.016769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017574.016773:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1840 00000100:00080000:2.0:1734017574.016813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017574.016817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017574.016823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017574.016833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334304768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017574.016843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017574.016846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017574.016848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017574.016851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1840 00000100:00080000:2.0:1734017574.016853:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017574.016856:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017574.016860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017574.016868:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334304896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017574.016878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017574.016881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017574.016882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017574.016886:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334304512:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017574.016886:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1840 00000100:00080000:2.0:1734017574.016888:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017574.016892:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017574.016896:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017574.016905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334305024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017574.016919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017574.016922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017574.016924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017574.016927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1840 00000100:00080000:2.0:1734017574.016929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017574.016933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017574.016937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017574.016944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334305152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017574.016952:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017574.016955:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017574.016956:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017574.016959:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1840 00000100:00080000:2.0:1734017574.016963:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017574.016966:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017574.016970:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017574.016977:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334305280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017574.016983:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017574.017010:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017574/real 1734017574] req@ffff8800a78a0700 x1818247334304512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017574.017031:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334304512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017574.017044:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017574.017048:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017574.017054:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334304512:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017574.017062:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334304640:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017574.017082:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017574.017089:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017574/real 1734017574] req@ffff8800a78a1880 x1818247334304640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017574.017101:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334304640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017574.017109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017574.017113:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017574.017117:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334304640:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017574.017124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334304768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017574.017139:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017574.017145:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017574/real 1734017574] req@ffff8800a78a3b80 x1818247334304768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017574.017156:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334304768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017574.017164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017574.017167:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017574.017170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334304768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017574.017201:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334304896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017574.017216:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017574.017222:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017574/real 1734017574] req@ffff8800a78a3480 x1818247334304896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017574.017233:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334304896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017574.017241:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017574.017244:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017574.017247:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334304896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017574.017253:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334305024:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017574.017270:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017574.017276:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017574/real 1734017574] req@ffff8800a78a1f80 x1818247334305024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017574.017287:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334305024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017574.017294:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017574.017297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017574.017300:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334305024:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017574.017305:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334305152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017574.017318:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017574.017324:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017574/real 1734017574] req@ffff8800a78a0a80 x1818247334305152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017574.017334:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334305152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017574.017341:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017574.017343:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017574.017346:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334305152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017574.017352:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334305280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017574.017365:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017574.017372:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017574/real 1734017574] req@ffff8800a78a2680 x1818247334305280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017574.017389:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334305280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017574.017397:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017574.017399:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017574.017402:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334305280:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017574.494815:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017575.496918:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017576.498623:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017577.500964:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017578.502734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017579.024601:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017579.024610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017579.024614:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017579.024625:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1845 00000100:00080000:2.0:1734017579.024634:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017579.024641:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017579.024679:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334305408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017579.024741:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017579.024745:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017579.024747:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017579.024752:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1845 00000100:00080000:2.0:1734017579.024755:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017579.024759:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017579.024770:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017579.024782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334305536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017579.024792:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017579.024796:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017579.024797:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017579.024802:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1845 00000100:00080000:2.0:1734017579.024804:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017579.024808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017579.024812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017579.024821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334305664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017579.024829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017579.024833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017579.024835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017579.024838:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1845 00000100:00080000:2.0:1734017579.024841:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017579.024844:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017579.024848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017579.024849:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334305408:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017579.024856:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334305792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017579.024865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017579.024868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017579.024869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017579.024873:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1845 00000100:00080000:2.0:1734017579.024875:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017579.024878:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017579.024882:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017579.024890:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334305920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017579.024898:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017579.024901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017579.024902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017579.024905:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1845 00000100:00080000:2.0:1734017579.024908:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017579.024911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017579.024915:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017579.024916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017579.024923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334306048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017579.024932:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017579.024935:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017579.024936:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017579/real 1734017579] req@ffff8800a78a2680 x1818247334305408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017579.024936:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017579.024940:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1845 00000100:00080000:2.0:1734017579.024943:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017579.024946:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017579.024950:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017579.024956:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334305408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017579.024957:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334306176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017579.024967:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017579.024970:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017579.024975:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334305408:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017579.024982:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334305536:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017579.024999:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017579.025005:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017579/real 1734017579] req@ffff8800a78a0a80 x1818247334305536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017579.025017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334305536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017579.025025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017579.025028:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017579.025032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334305536:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017579.025038:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334305664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017579.025052:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017579.025058:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017579/real 1734017579] req@ffff8800a78a1f80 x1818247334305664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017579.025068:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334305664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017579.025076:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017579.025078:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017579.025082:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334305664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017579.025106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334305792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017579.025118:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017579.025124:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017579/real 1734017579] req@ffff8800a78a3480 x1818247334305792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017579.025135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334305792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017579.025143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017579.025145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017579.025148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334305792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017579.025153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334305920:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017579.025169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017579.025175:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017579/real 1734017579] req@ffff8800a78a3b80 x1818247334305920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017579.025186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334305920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017579.025193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017579.025195:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017579.025198:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334305920:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017579.025203:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334306048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017579.025214:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017579.025220:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017579/real 1734017579] req@ffff8800a78a1880 x1818247334306048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017579.025230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334306048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017579.025237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017579.025239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017579.025242:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334306048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017579.025248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334306176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017579.025260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017579.025266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017579/real 1734017579] req@ffff8800a78a0700 x1818247334306176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017579.025282:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334306176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017579.025289:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017579.025291:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017579.025294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334306176:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734017579.504786:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017580.506735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017581.508826:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017582.510806:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017583.512801:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017584.032644:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017584.032655:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017584.032661:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017584.032673:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1850 00000100:00080000:2.0:1734017584.032684:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017584.032693:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017584.032741:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334306304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017584.032780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017584.032787:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017584.032789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017584.032794:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1850 00000100:00080000:2.0:1734017584.032798:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017584.032802:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017584.032812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017584.032824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334306432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017584.032834:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017584.032838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017584.032839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017584.032843:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1850 00000100:00080000:2.0:1734017584.032846:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017584.032850:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017584.032855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017584.032865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334306560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017584.032874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017584.032877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017584.032879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017584.032883:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1850 00000100:00080000:2.0:1734017584.032885:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017584.032889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017584.032894:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017584.032905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334306688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017584.032907:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334306304:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017584.032915:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017584.032918:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017584.032920:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017584.032923:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1850 00000100:00080000:2.0:1734017584.032926:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017584.032929:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017584.032933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017584.032942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334306816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017584.032950:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017584.032953:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017584.032954:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017584.032957:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1850 00000100:00080000:2.0:1734017584.032960:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017584.032964:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017584.032968:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017584.032975:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334306944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017584.032981:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017584.032983:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017584.032986:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017584.032988:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017584.032993:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1850 00000100:00080000:2.0:1734017584.032999:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017584.033002:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017584.033004:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017584/real 1734017584] req@ffff8800a78a0700 x1818247334306304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734017584.033007:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017584.033015:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334307072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017584.033024:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334306304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017584.033036:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017584.033040:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017584.033045:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334306304:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017584.033053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334306432:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017584.033072:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017584.033079:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017584/real 1734017584] req@ffff8800a78a1880 x1818247334306432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017584.033090:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334306432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017584.033098:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017584.033102:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017584.033105:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334306432:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017584.033112:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334306560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017584.033127:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017584.033133:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017584/real 1734017584] req@ffff8800a78a3b80 x1818247334306560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017584.033143:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334306560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017584.033151:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017584.033154:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017584.033157:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334306560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017584.033184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334306688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017584.033198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017584.033204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017584/real 1734017584] req@ffff8800a78a3480 x1818247334306688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017584.033214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334306688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017584.033222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017584.033225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017584.033228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334306688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017584.033234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334306816:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017584.033250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017584.033257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017584/real 1734017584] req@ffff8800a78a1f80 x1818247334306816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017584.033267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334306816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017584.033275:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017584.033277:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017584.033280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334306816:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017584.033286:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334306944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017584.033298:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017584.033304:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017584/real 1734017584] req@ffff8800a78a0a80 x1818247334306944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017584.033313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334306944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017584.033320:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017584.033323:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017584.033326:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334306944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017584.033331:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334307072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017584.033344:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017584.033350:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017584/real 1734017584] req@ffff8800a78a2680 x1818247334307072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017584.033367:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334307072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017584.033374:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017584.033377:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017584.033379:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334307072:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017584.514793:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017585.516764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017586.518790:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017587.520799:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017588.522737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017589.040575:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017589.040582:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017589.040586:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017589.040594:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1855 00000100:00080000:2.0:1734017589.040600:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017589.040605:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017589.040633:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334307200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017589.040655:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017589.040658:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017589.040659:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017589.040662:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1855 00000100:00080000:2.0:1734017589.040664:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017589.040666:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017589.040673:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017589.040680:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334307328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017589.040687:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017589.040689:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017589.040690:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017589.040692:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1855 00000100:00080000:2.0:1734017589.040694:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017589.040696:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017589.040699:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017589.040704:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334307456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017589.040709:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017589.040711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017589.040712:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017589.040713:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1855 00000100:00080000:2.0:1734017589.040715:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017589.040717:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017589.040719:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017589.040724:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334307584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017589.040730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017589.040731:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017589.040732:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017589.040734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1855 00000100:00100000:1.0:1734017589.040735:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334307200:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017589.040736:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017589.040738:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017589.040741:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017589.040745:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334307712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017589.040750:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017589.040752:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017589.040753:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017589.040755:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1855 00000100:00080000:2.0:1734017589.040756:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017589.040758:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017589.040760:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017589.040765:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334307840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017589.040769:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017589.040771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017589.040772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017589.040774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1855 00000100:00080000:2.0:1734017589.040775:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017589.040777:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017589.040779:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017589.040783:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017589.040784:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334307968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017589.040797:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017589/real 1734017589] req@ffff8800a78a2680 x1818247334307200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017589.040810:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334307200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017589.040816:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017589.040818:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017589.040821:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334307200:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017589.040826:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334307328:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017589.040836:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017589.040840:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017589/real 1734017589] req@ffff8800a78a0a80 x1818247334307328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017589.040847:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334307328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017589.040851:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017589.040853:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017589.040855:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334307328:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017589.040859:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334307456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017589.040868:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017589.040872:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017589/real 1734017589] req@ffff8800a78a1f80 x1818247334307456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017589.040877:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334307456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017589.040882:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017589.040884:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017589.040886:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334307456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017589.040902:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334307584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017589.040909:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017589.040913:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017589/real 1734017589] req@ffff8800a78a3480 x1818247334307584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017589.040919:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334307584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017589.040924:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017589.040925:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017589.040927:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334307584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017589.040930:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334307712:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017589.040940:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017589.040943:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017589/real 1734017589] req@ffff8800a78a3b80 x1818247334307712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017589.040949:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334307712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017589.040954:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017589.040955:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017589.040956:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334307712:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017589.040960:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334307840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017589.040966:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017589.040970:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017589/real 1734017589] req@ffff8800a78a1880 x1818247334307840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017589.040975:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334307840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017589.040979:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017589.040980:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017589.040982:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334307840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017589.040985:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334307968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017589.040993:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017589.040996:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017589/real 1734017589] req@ffff8800a78a0700 x1818247334307968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017589.041006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334307968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017589.041011:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017589.041012:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017589.041014:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334307968:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734017589.524765:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017590.526757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017591.528702:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017592.530668:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017593.532709:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017594.048572:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017594.048581:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017594.048586:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017594.048596:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1860 00000100:00080000:2.0:1734017594.048605:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017594.048612:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017594.048651:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334308096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017594.048687:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017594.048691:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017594.048693:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017594.048697:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1860 00000100:00080000:2.0:1734017594.048701:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017594.048705:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017594.048715:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017594.048727:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334308224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017594.048736:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017594.048739:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017594.048741:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017594.048745:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1860 00000100:00080000:2.0:1734017594.048748:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017594.048752:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017594.048757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017594.048766:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334308352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017594.048774:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017594.048777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017594.048779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017594.048782:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1860 00000100:00080000:2.0:1734017594.048784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017594.048787:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017594.048791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017594.048799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334308480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017594.048806:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334308096:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017594.048808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017594.048811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017594.048812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017594.048816:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1860 00000100:00080000:2.0:1734017594.048818:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017594.048821:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017594.048825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017594.048831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334308608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017594.048846:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017594.048849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017594.048850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017594.048854:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1860 00000100:00080000:2.0:1734017594.048856:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017594.048860:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017594.048864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017594.048870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334308736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017594.048874:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017594.048878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017594.048881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017594.048883:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017594.048886:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1860 00000100:00080000:2.0:1734017594.048890:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017594.048894:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017594.048895:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017594/real 1734017594] req@ffff8800a78a0700 x1818247334308096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734017594.048898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017594.048905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334308864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017594.048915:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334308096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017594.048926:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017594.048929:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017594.048934:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334308096:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017594.048941:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334308224:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017594.048958:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017594.048965:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017594/real 1734017594] req@ffff8800a78a1880 x1818247334308224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017594.048976:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334308224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017594.048984:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017594.048987:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017594.048991:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334308224:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017594.048997:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334308352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017594.049011:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017594.049017:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017594/real 1734017594] req@ffff8800a78a3b80 x1818247334308352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017594.049027:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334308352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017594.049034:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017594.049037:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017594.049041:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334308352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017594.049065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334308480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017594.049078:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017594.049084:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017594/real 1734017594] req@ffff8800a78a3480 x1818247334308480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017594.049094:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334308480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017594.049102:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017594.049104:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017594.049107:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334308480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017594.049113:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334308608:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017594.049129:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017594.049135:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017594/real 1734017594] req@ffff8800a78a1f80 x1818247334308608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017594.049146:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334308608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017594.049153:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017594.049155:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017594.049158:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334308608:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017594.049163:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334308736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017594.049175:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017594.049181:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017594/real 1734017594] req@ffff8800a78a0a80 x1818247334308736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017594.049191:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334308736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017594.049198:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017594.049200:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017594.049203:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334308736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017594.049208:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334308864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017594.049221:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017594.049227:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017594/real 1734017594] req@ffff8800a78a2680 x1818247334308864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017594.049243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334308864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017594.049250:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017594.049252:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017594.049255:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334308864:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017594.534776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017595.536728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017596.538722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017597.540760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017598.542746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017599.056614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017599.056623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017599.056628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017599.056639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1865 00000100:00080000:2.0:1734017599.056647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017599.056655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017599.056694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334308992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017599.056728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017599.056732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017599.056734:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017599.056739:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1865 00000100:00080000:2.0:1734017599.056742:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017599.056747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017599.056757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017599.056768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334309120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017599.056778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017599.056781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017599.056783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017599.056787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1865 00000100:00080000:2.0:1734017599.056790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017599.056794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017599.056798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017599.056807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334309248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017599.056815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017599.056819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017599.056821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017599.056825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1865 00000100:00080000:2.0:1734017599.056827:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017599.056830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017599.056834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017599.056841:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334308992:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017599.056842:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334309376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017599.056851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017599.056854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017599.056856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017599.056859:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1865 00000100:00080000:2.0:1734017599.056861:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017599.056864:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017599.056868:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017599.056876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334309504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017599.056884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017599.056886:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017599.056888:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017599.056891:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1865 00000100:00080000:2.0:1734017599.056903:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017599.056907:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017599.056912:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017599.056920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334309632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017599.056928:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017599.056931:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017599.056932:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017599.056933:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334309120:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017599.056936:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1865 00000100:00080000:2.0:1734017599.056938:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017599.056941:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017599.056945:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017599.056953:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334309760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017599.056954:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334309248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017599.056978:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334309376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017599.056992:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334309504:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017599.057007:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334309632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017599.057022:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334309760:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734017599.544782:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017600.546734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017601.548744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017602.062729:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017602.062776:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017602.062795:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017602.062835:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017602.062843:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017602.062850:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017602.062856:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017602.062862:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017602.062867:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734017602.062961:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017599/real 1734017602] req@ffff8800a78a3480 x1818247334309376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017602.062998:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334309376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017602.063016:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017602.063022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017602.063029:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334309376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017602.063042:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017599/real 1734017602] req@ffff8800a78a3b80 x1818247334309504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017602.063061:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334309504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017602.063075:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017602.063079:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017602.063085:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334309504:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017602.063096:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017599/real 1734017602] req@ffff8800a78a1880 x1818247334309632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017602.063112:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334309632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017602.063124:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017602.063128:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017602.063133:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334309632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017602.063143:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017599/real 1734017602] req@ffff8800a78a0700 x1818247334309760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017602.063161:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334309760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017602.063173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017602.063177:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017602.063183:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334309760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017602.063198:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017599/real 1734017602] req@ffff8800a78a2680 x1818247334308992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017602.063218:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334308992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017602.063230:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017602.063235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017602.063240:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334308992:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017602.063251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017599/real 1734017602] req@ffff8800a78a0a80 x1818247334309120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017602.063269:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334309120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017602.063281:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017602.063285:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017602.063290:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334309120:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017602.063343:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017599/real 1734017602] req@ffff8800a78a1f80 x1818247334309248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017602.063364:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334309248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017602.063376:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017602.063380:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017602.063385:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334309248:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017602.550739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017603.552746:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017604.064614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017604.064623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017604.064628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017604.064638:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1870 00000100:00080000:2.0:1734017604.064647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017604.064655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017604.064693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334309888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017604.064729:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017604.064735:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017604.064737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017604.064742:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1870 00000100:00080000:2.0:1734017604.064745:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017604.064749:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017604.064758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017604.064770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334310016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017604.064780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017604.064783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017604.064785:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017604.064789:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1870 00000100:00080000:2.0:1734017604.064792:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017604.064795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017604.064800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017604.064809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334310144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017604.064817:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017604.064820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017604.064822:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017604.064825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1870 00000100:00080000:2.0:1734017604.064827:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017604.064830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017604.064835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017604.064843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334310272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017604.064844:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334309888:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017604.064853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017604.064855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017604.064857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017604.064860:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1870 00000100:00080000:2.0:1734017604.064863:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017604.064866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017604.064870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017604.064877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334310400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017604.064885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017604.064888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017604.064889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017604.064892:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1870 00000100:00080000:2.0:1734017604.064896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017604.064899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017604.064903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017604.064911:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017604.064911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334310528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017604.064919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017604.064922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017604.064924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017604.064927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1870 00000100:00080000:2.0:1734017604.064930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017604.064931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017604/real 1734017604] req@ffff8800a78a1f80 x1818247334309888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017604.064933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017604.064937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017604.064945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334310656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017604.064951:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334309888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017604.064961:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017604.064964:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017604.064969:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334309888:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017604.064977:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334310016:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017604.064993:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017604.065000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017604/real 1734017604] req@ffff8800a78a0a80 x1818247334310016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017604.065012:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334310016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017604.065019:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017604.065022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017604.065026:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334310016:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017604.065031:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334310144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017604.065046:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017604.065052:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017604/real 1734017604] req@ffff8800a78a2680 x1818247334310144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017604.065062:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334310144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017604.065070:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017604.065073:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017604.065076:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334310144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017604.065107:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334310272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017604.065120:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017604.065125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017604/real 1734017604] req@ffff8800a78a0700 x1818247334310272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017604.065136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334310272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017604.065144:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017604.065146:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017604.065149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334310272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017604.065154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334310400:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017604.065166:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017604.065172:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017604/real 1734017604] req@ffff8800a78a1880 x1818247334310400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017604.065182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334310400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017604.065189:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017604.065192:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017604.065195:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334310400:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017604.065201:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334310528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017604.065213:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017604.065219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017604/real 1734017604] req@ffff8800a78a3b80 x1818247334310528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017604.065229:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334310528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017604.065236:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017604.065238:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017604.065241:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334310528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017604.065247:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334310656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017604.065259:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017604.065318:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017604/real 1734017604] req@ffff8800a78a3480 x1818247334310656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017604.065330:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334310656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017604.065337:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017604.065339:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017604.065342:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334310656:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017604.554733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017605.556736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017606.558729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017607.560717:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017608.562743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017609.072621:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017609.072630:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017609.072634:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017609.072645:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1875 00000100:00080000:2.0:1734017609.072654:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017609.072662:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017609.072699:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334310784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017609.072735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017609.072739:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017609.072741:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017609.072746:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1875 00000100:00080000:2.0:1734017609.072749:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017609.072753:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017609.072763:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017609.072775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334310912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017609.072785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017609.072788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017609.072790:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017609.072794:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1875 00000100:00080000:2.0:1734017609.072797:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017609.072800:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017609.072805:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017609.072814:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334311040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017609.072822:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017609.072825:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017609.072827:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017609.072830:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1875 00000100:00080000:2.0:1734017609.072832:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017609.072835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017609.072840:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017609.072843:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334310784:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017609.072848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334311168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017609.072857:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017609.072860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017609.072861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017609.072865:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1875 00000100:00080000:2.0:1734017609.072867:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017609.072870:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017609.072875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017609.072882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334311296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017609.072890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017609.072893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017609.072894:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017609.072897:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1875 00000100:00080000:2.0:1734017609.072900:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017609.072904:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017609.072907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017609.072909:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017609.072915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334311424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017609.072923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017609.072925:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017609.072927:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017609.072930:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1875 00000100:00100000:1.0:1734017609.072931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017609/real 1734017609] req@ffff8800a78a3480 x1818247334310784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017609.072933:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017609.072936:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017609.072940:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017609.072947:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334311552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017609.072951:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334310784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017609.072962:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017609.072966:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017609.072970:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334310784:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017609.072978:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334310912:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017609.072994:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017609.073001:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017609/real 1734017609] req@ffff8800a78a3b80 x1818247334310912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017609.073012:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334310912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017609.073020:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017609.073022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017609.073026:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334310912:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017609.073031:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334311040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017609.073045:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017609.073051:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017609/real 1734017609] req@ffff8800a78a1880 x1818247334311040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017609.073061:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334311040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017609.073069:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017609.073072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017609.073075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334311040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017609.073105:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334311168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017609.073119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017609.073125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017609/real 1734017609] req@ffff8800a78a0700 x1818247334311168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017609.073135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334311168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017609.073142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017609.073144:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017609.073147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334311168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017609.073153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334311296:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017609.073165:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017609.073171:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017609/real 1734017609] req@ffff8800a78a2680 x1818247334311296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017609.073213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334311296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017609.073221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017609.073223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017609.073226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334311296:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017609.073233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334311424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017609.073252:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017609.073259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017609/real 1734017609] req@ffff8800a78a0a80 x1818247334311424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017609.073269:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334311424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017609.073276:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017609.073278:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017609.073281:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334311424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017609.073287:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334311552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017609.073299:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017609.073306:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017609/real 1734017609] req@ffff8800a78a1f80 x1818247334311552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017609.073316:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334311552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017609.073323:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017609.073325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017609.073328:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334311552:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734017609.564771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017610.566718:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017611.568761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017612.570718:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017613.572719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017614.080610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017614.080619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017614.080624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017614.080634:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1880 00000100:00080000:2.0:1734017614.080643:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017614.080650:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017614.080689:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334311680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017614.080721:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017614.080726:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017614.080727:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017614.080732:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1880 00000100:00080000:2.0:1734017614.080735:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017614.080739:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017614.080749:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017614.080761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334311808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017614.080770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017614.080774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017614.080775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017614.080780:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1880 00000100:00080000:2.0:1734017614.080783:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017614.080787:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017614.080791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017614.080800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334311936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017614.080809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017614.080812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017614.080813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017614.080816:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1880 00000100:00080000:2.0:1734017614.080819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017614.080822:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017614.080826:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017614.080827:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334311680:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017614.080834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334312064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017614.080843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017614.080845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017614.080847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017614.080850:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1880 00000100:00080000:2.0:1734017614.080852:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017614.080855:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017614.080859:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017614.080867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334312192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017614.080881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017614.080884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017614.080886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017614.080889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1880 00000100:00080000:2.0:1734017614.080892:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017614.080893:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017614.080895:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017614.080899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017614.080907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334312320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017614.080914:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017614/real 1734017614] req@ffff8800a78a1f80 x1818247334311680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017614.080914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017614.080917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017614.080919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017614.080922:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1880 00000100:00080000:2.0:1734017614.080925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017614.080928:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017614.080932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017614.080939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334312448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017614.080943:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334311680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017614.080953:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017614.080956:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017614.080961:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334311680:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017614.080968:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334311808:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017614.080985:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017614.080992:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017614/real 1734017614] req@ffff8800a78a0a80 x1818247334311808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017614.081003:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334311808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017614.081010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017614.081013:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017614.081016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334311808:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017614.081022:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334311936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017614.081036:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017614.081042:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017614/real 1734017614] req@ffff8800a78a2680 x1818247334311936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017614.081052:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334311936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017614.081059:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017614.081062:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017614.081065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334311936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017614.081090:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334312064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017614.081142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017614.081149:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017614/real 1734017614] req@ffff8800a78a0700 x1818247334312064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017614.081160:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334312064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017614.081168:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017614.081170:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017614.081173:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334312064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017614.081179:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334312192:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017614.081191:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017614.081197:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017614/real 1734017614] req@ffff8800a78a1880 x1818247334312192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017614.081208:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334312192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017614.081215:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017614.081217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017614.081220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334312192:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017614.081226:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334312320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017614.081240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017614.081247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017614/real 1734017614] req@ffff8800a78a3b80 x1818247334312320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017614.081256:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334312320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017614.081263:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017614.081265:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017614.081268:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334312320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017614.081273:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334312448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017614.081285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017614.081291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017614/real 1734017614] req@ffff8800a78a3480 x1818247334312448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017614.081301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334312448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017614.081308:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017614.081310:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017614.081313:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334312448:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017614.574777:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017615.576724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017616.578712:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017617.580717:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017618.582716:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017619.088619:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017619.088628:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017619.088633:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017619.088644:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1885 00000100:00080000:2.0:1734017619.088652:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017619.088660:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017619.088698:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334312576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017619.088733:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017619.088737:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017619.088739:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017619.088744:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1885 00000100:00080000:2.0:1734017619.088747:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017619.088751:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017619.088761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017619.088773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334312704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017619.088783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017619.088786:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017619.088788:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017619.088792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1885 00000100:00080000:2.0:1734017619.088794:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017619.088798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017619.088802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017619.088812:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334312832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017619.088820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017619.088824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017619.088826:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017619.088829:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1885 00000100:00080000:2.0:1734017619.088832:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017619.088835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017619.088839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017619.088845:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334312576:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017619.088847:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334312960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017619.088856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017619.088859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017619.088861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017619.088864:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1885 00000100:00080000:2.0:1734017619.088867:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017619.088870:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017619.088874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017619.088881:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334313088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017619.088889:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017619.088892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017619.088893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017619.088897:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1885 00000100:00080000:2.0:1734017619.088900:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017619.088903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017619.088907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017619.088910:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017619.088915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334313216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017619.088923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017619.088927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017619.088928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017619.088931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017619/real 1734017619] req@ffff8800a78a3480 x1818247334312576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017619.088931:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1885 00000100:00080000:2.0:1734017619.088934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017619.088937:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017619.088941:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017619.088948:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334313344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017619.088958:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334312576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017619.088969:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017619.088972:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017619.088976:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334312576:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017619.088984:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334312704:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017619.089000:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017619.089007:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017619/real 1734017619] req@ffff8800a78a3b80 x1818247334312704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017619.089054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334312704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017619.089062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017619.089065:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017619.089068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334312704:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017619.089075:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334312832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017619.089092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017619.089099:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017619/real 1734017619] req@ffff8800a78a1880 x1818247334312832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017619.089110:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334312832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017619.089118:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017619.089121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017619.089124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334312832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017619.089148:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334312960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017619.089162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017619.089168:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017619/real 1734017619] req@ffff8800a78a0700 x1818247334312960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017619.089178:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334312960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017619.089186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017619.089188:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017619.089191:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334312960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017619.089197:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334313088:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017619.089209:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017619.089215:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017619/real 1734017619] req@ffff8800a78a2680 x1818247334313088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017619.089225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334313088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017619.089232:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017619.089234:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017619.089237:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334313088:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017619.089243:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334313216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017619.089258:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017619.089264:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017619/real 1734017619] req@ffff8800a78a0a80 x1818247334313216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017619.089274:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334313216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017619.089281:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017619.089283:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017619.089286:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334313216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017619.089291:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334313344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017619.089303:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017619.089309:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017619/real 1734017619] req@ffff8800a78a1f80 x1818247334313344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017619.089319:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334313344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017619.089325:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017619.089328:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017619.089330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334313344:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734017619.584743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017620.586726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017621.588739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017622.590732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017623.592736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017624.096592:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017624.096600:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017624.096605:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017624.096615:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1890 00000100:00080000:2.0:1734017624.096623:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017624.096631:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017624.096669:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334313472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017624.096705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017624.096711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017624.096713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017624.096717:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1890 00000100:00080000:2.0:1734017624.096721:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017624.096724:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017624.096735:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017624.096746:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334313600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017624.096756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017624.096759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017624.096760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017624.096764:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1890 00000100:00080000:2.0:1734017624.096767:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017624.096771:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017624.096776:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017624.096784:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334313728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017624.096793:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017624.096796:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017624.096797:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017624.096801:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1890 00000100:00080000:2.0:1734017624.096803:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017624.096807:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017624.096811:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017624.096820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334313856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017624.096821:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334313472:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017624.096829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017624.096832:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017624.096833:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017624.096836:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1890 00000100:00080000:2.0:1734017624.096839:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017624.096842:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017624.096846:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017624.096853:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334313984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017624.096861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017624.096864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017624.096865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017624.096869:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1890 00000100:00080000:2.0:1734017624.096872:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017624.096875:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017624.096879:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017624.096887:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017624.096887:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334314112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017624.096895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017624.096898:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017624.096901:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017624.096904:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1890 00000100:00080000:2.0:1734017624.096907:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017624.096908:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017624/real 1734017624] req@ffff8800a78a1f80 x1818247334313472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017624.096910:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017624.096914:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017624.096921:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334314240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017624.096963:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334313472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017624.096974:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017624.096977:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017624.096982:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334313472:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017624.096990:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334313600:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017624.097010:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017624.097017:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017624/real 1734017624] req@ffff8800a78a0a80 x1818247334313600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017624.097028:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334313600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017624.097036:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017624.097038:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017624.097042:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334313600:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017624.097047:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334313728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017624.097060:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017624.097066:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017624/real 1734017624] req@ffff8800a78a2680 x1818247334313728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017624.097077:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334313728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017624.097084:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017624.097087:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017624.097090:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334313728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017624.097114:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334313856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017624.097127:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017624.097134:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017624/real 1734017624] req@ffff8800a78a0700 x1818247334313856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017624.097144:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334313856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017624.097151:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017624.097154:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017624.097157:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334313856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017624.097162:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334313984:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017624.097174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017624.097180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017624/real 1734017624] req@ffff8800a78a1880 x1818247334313984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017624.097190:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334313984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017624.097197:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017624.097199:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017624.097203:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334313984:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017624.097208:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334314112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017624.097223:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017624.097230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017624/real 1734017624] req@ffff8800a78a3b80 x1818247334314112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017624.097240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334314112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017624.097247:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017624.097249:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017624.097252:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334314112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017624.097257:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334314240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017624.097269:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017624.097275:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017624/real 1734017624] req@ffff8800a78a3480 x1818247334314240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017624.097285:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334314240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017624.097292:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017624.097294:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017624.097297:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334314240:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017624.594722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017625.596728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017626.598755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017627.600718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017628.602726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017629.104598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017629.104607:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017629.104611:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017629.104622:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1895 00000100:00080000:2.0:1734017629.104631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017629.104639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017629.104677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334314368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017629.104712:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017629.104716:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017629.104718:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017629.104722:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1895 00000100:00080000:2.0:1734017629.104725:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017629.104729:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017629.104739:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017629.104751:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334314496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017629.104762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017629.104765:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017629.104767:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017629.104771:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1895 00000100:00080000:2.0:1734017629.104773:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017629.104777:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017629.104781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017629.104791:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334314624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017629.104799:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017629.104802:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017629.104803:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017629.104807:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1895 00000100:00080000:2.0:1734017629.104809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017629.104812:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017629.104816:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017629.104822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334314368:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017629.104824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334314752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017629.104833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017629.104836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017629.104868:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017629.104873:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1895 00000100:00080000:2.0:1734017629.104876:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017629.104879:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017629.104884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017629.104893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334314880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017629.104902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017629.104905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017629.104906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017629.104910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1895 00000100:00080000:2.0:1734017629.104913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017629.104917:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017629.104920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017629.104924:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017629.104928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334315008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017629.104936:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017629.104939:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017629.104940:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017629.104943:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1895 00000100:00100000:1.0:1734017629.104946:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017629/real 1734017629] req@ffff8800a78a3480 x1818247334314368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017629.104946:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017629.104950:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017629.104954:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017629.104962:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334315136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017629.104974:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334314368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017629.104984:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017629.104987:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017629.104992:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334314368:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017629.104999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334314496:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017629.105017:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017629.105024:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017629/real 1734017629] req@ffff8800a78a3b80 x1818247334314496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017629.105035:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334314496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017629.105042:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017629.105045:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017629.105048:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334314496:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017629.105054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334314624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017629.105067:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017629.105073:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017629/real 1734017629] req@ffff8800a78a1880 x1818247334314624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017629.105083:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334314624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017629.105090:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017629.105093:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017629.105096:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334314624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017629.105120:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334314752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017629.105133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017629.105140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017629/real 1734017629] req@ffff8800a78a0700 x1818247334314752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017629.105150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334314752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017629.105157:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017629.105159:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017629.105162:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334314752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017629.105168:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334314880:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017629.105180:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017629.105186:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017629/real 1734017629] req@ffff8800a78a2680 x1818247334314880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017629.105196:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334314880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017629.105203:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017629.105205:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017629.105208:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334314880:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017629.105213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334315008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017629.105228:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017629.105235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017629/real 1734017629] req@ffff8800a78a0a80 x1818247334315008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017629.105245:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334315008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017629.105252:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017629.105254:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017629.105257:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334315008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017629.105262:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334315136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017629.105275:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017629.105281:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017629/real 1734017629] req@ffff8800a78a1f80 x1818247334315136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017629.105290:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334315136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017629.105297:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017629.105299:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017629.105302:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334315136:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734017629.604730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017630.606755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017631.608728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017632.610751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017633.612719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017634.112591:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017634.112601:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017634.112605:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017634.112616:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1900 00000100:00080000:2.0:1734017634.112624:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017634.112632:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017634.112671:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334315264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017634.112705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017634.112709:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017634.112711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017634.112716:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1900 00000100:00080000:2.0:1734017634.112719:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017634.112723:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017634.112733:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017634.112745:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334315392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017634.112781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017634.112784:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017634.112786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017634.112791:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1900 00000100:00080000:2.0:1734017634.112795:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017634.112798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017634.112804:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017634.112814:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334315520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017634.112824:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017634.112827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017634.112829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017634.112832:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1900 00000100:00080000:2.0:1734017634.112835:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017634.112837:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334315264:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017634.112838:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017634.112842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017634.112850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334315648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017634.112858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017634.112861:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017634.112863:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017634.112866:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1900 00000100:00080000:2.0:1734017634.112868:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017634.112871:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017634.112876:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017634.112883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334315776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017634.112897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017634.112900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017634.112902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017634.112904:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017634.112905:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1900 00000100:00080000:2.0:1734017634.112908:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017634.112911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017634.112915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017634.112923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334315904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017634.112926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017634/real 1734017634] req@ffff8800a78a1f80 x1818247334315264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017634.112931:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017634.112934:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017634.112935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017634.112939:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1900 00000100:00080000:2.0:1734017634.112941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017634.112944:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017634.112948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017634.112953:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334315264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017634.112956:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334316032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017634.112964:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017634.112967:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017634.112972:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334315264:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017634.112979:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334315392:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017634.112996:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017634.113003:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017634/real 1734017634] req@ffff8800a78a0a80 x1818247334315392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017634.113014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334315392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017634.113022:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017634.113024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017634.113028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334315392:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017634.113050:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334315520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017634.113064:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017634.113070:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017634/real 1734017634] req@ffff8800a78a2680 x1818247334315520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017634.113080:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334315520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017634.113088:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017634.113091:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017634.113094:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334315520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017634.113100:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334315648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017634.113113:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017634.113118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017634/real 1734017634] req@ffff8800a78a0700 x1818247334315648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017634.113128:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334315648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017634.113135:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017634.113138:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017634.113140:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334315648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017634.113146:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334315776:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017634.113158:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017634.113164:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017634/real 1734017634] req@ffff8800a78a1880 x1818247334315776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017634.113174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334315776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017634.113181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017634.113183:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017634.113187:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334315776:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017634.113192:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334315904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017634.113207:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017634.113214:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017634/real 1734017634] req@ffff8800a78a3b80 x1818247334315904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017634.113224:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334315904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017634.113231:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017634.113233:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017634.113236:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334315904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017634.113241:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334316032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017634.113253:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017634.113259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017634/real 1734017634] req@ffff8800a78a3480 x1818247334316032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017634.113269:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334316032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017634.113276:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017634.113278:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017634.113281:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334316032:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017634.614786:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017635.616689:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017636.618772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017637.620763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017638.622741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017639.120595:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017639.120604:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017639.120609:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017639.120619:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1905 00000100:00080000:2.0:1734017639.120628:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017639.120636:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017639.120710:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334316160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017639.120743:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017639.120747:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017639.120749:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017639.120754:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1905 00000100:00080000:2.0:1734017639.120757:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017639.120761:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017639.120771:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017639.120784:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334316288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017639.120793:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017639.120797:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017639.120798:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017639.120802:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1905 00000100:00080000:2.0:1734017639.120805:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017639.120809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017639.120813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017639.120822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334316416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017639.120830:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017639.120834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017639.120836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017639.120839:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1905 00000100:00080000:2.0:1734017639.120842:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017639.120845:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017639.120849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017639.120857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334316544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017639.120859:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334316160:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017639.120866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017639.120869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017639.120871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017639.120874:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1905 00000100:00080000:2.0:1734017639.120876:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017639.120879:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017639.120883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017639.120891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334316672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017639.120899:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017639.120901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017639.120903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017639.120906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1905 00000100:00080000:2.0:1734017639.120909:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017639.120913:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017639.120918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017639.120925:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017639.120926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334316800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017639.120934:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017639.120937:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017639.120938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017639.120942:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1905 00000100:00080000:2.0:1734017639.120944:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017639.120946:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017639/real 1734017639] req@ffff8800a78a3480 x1818247334316160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017639.120947:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017639.120951:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017639.120958:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334316928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017639.120974:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334316160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017639.120984:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017639.120987:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017639.120992:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334316160:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017639.120999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334316288:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017639.121016:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017639.121023:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017639/real 1734017639] req@ffff8800a78a3b80 x1818247334316288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017639.121034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334316288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017639.121041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017639.121043:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017639.121047:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334316288:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017639.121053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334316416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017639.121067:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017639.121073:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017639/real 1734017639] req@ffff8800a78a1880 x1818247334316416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017639.121083:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334316416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017639.121090:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017639.121093:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017639.121096:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334316416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017639.121120:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334316544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017639.121133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017639.121139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017639/real 1734017639] req@ffff8800a78a0700 x1818247334316544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017639.121150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334316544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017639.121157:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017639.121159:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017639.121162:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334316544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017639.121168:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334316672:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017639.121180:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017639.121186:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017639/real 1734017639] req@ffff8800a78a2680 x1818247334316672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017639.121196:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334316672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017639.121203:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017639.121205:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017639.121208:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334316672:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017639.121214:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334316800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017639.121229:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017639.121235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017639/real 1734017639] req@ffff8800a78a0a80 x1818247334316800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017639.121245:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334316800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017639.121252:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017639.121254:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017639.121257:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334316800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017639.121262:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334316928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017639.121275:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017639.121281:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017639/real 1734017639] req@ffff8800a78a1f80 x1818247334316928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017639.121290:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334316928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017639.121297:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017639.121299:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017639.121302:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334316928:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017639.624830:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017640.626698:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017641.628699:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017642.630772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017643.632773:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017644.128750:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017644.128763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017644.128768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017644.128778:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1910 00000100:00080000:0.0:1734017644.128787:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017644.128794:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017644.128831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334317056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017644.128880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017644.128886:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017644.128888:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017644.128895:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1910 00000100:00080000:0.0:1734017644.128900:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017644.128906:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017644.128919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017644.128937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334317184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017644.128948:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334317056:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734017644.128953:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017644.128957:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017644.128959:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017644.128965:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1910 00000100:00080000:0.0:1734017644.128969:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017644.128974:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017644.128980:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017644.128992:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334317312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017644.129004:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017644.129008:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017644.129010:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017644.129015:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1910 00000800:00000100:1.0:1734017644.129018:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734017644.129019:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017644.129024:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017644.129030:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017644.129042:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334317440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017644.129042:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017644/real 1734017644] req@ffff8800a934c380 x1818247334317056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017644.129056:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017644.129060:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017644.129063:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017644.129068:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1910 00000100:00080000:0.0:1734017644.129071:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017644.129076:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017644.129077:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334317056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734017644.129083:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734017644.129091:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1734017644.129095:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334317568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017644.129096:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017644.129103:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334317056:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734017644.129109:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017644.129113:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017644.129115:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017644.129120:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1910 00000100:00080000:0.0:1734017644.129123:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017644.129128:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017644.129130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334317184:192.168.201.116@tcp:38:kworker.0 00000008:00100000:0.0:1734017644.129134:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017644.129146:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334317696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017644.129155:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734017644.129161:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017644.129166:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017644/real 1734017644] req@ffff8800a934ea00 x1818247334317184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017644.129167:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017644.129170:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017644.129175:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1910 00000100:00080000:0.0:1734017644.129179:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017644.129184:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017644.129184:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334317184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734017644.129191:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017644.129196:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017644.129200:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734017644.129203:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334317824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017644.129206:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334317184:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017644.129215:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334317312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017644.129238:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017644.129250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017644/real 1734017644] req@ffff8800a934ce00 x1818247334317312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017644.129267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334317312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017644.129279:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017644.129289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017644.129294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334317312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017644.129301:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334317440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017644.129321:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017644.129329:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017644/real 1734017644] req@ffff8800a934df80 x1818247334317440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017644.129340:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334317440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017644.129347:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017644.129350:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017644.129353:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334317440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017644.129358:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334317568:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017644.129371:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017644.129377:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017644/real 1734017644] req@ffff8800a934d500 x1818247334317568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017644.129387:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334317568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017644.129394:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017644.129396:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017644.129399:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334317568:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017644.129417:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334317696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017644.129433:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017644.129439:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017644/real 1734017644] req@ffff8800a934fb80 x1818247334317696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017644.129449:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334317696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017644.129457:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017644.129459:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017644.129462:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334317696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017644.129727:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334317824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017644.129748:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017644.129756:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017644/real 1734017644] req@ffff8800a934f100 x1818247334317824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017644.129767:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334317824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017644.129775:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017644.129777:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017644.129780:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334317824:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017644.634776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017645.636770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017646.638748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017647.640750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017648.642743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017649.136589:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017649.136595:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017649.136598:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017649.136605:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1915 00000100:00080000:2.0:1734017649.136611:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017649.136616:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017649.136644:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334317952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017649.136669:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017649.136671:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017649.136672:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017649.136675:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1915 00000100:00080000:2.0:1734017649.136677:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017649.136680:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017649.136687:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017649.136694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334318080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017649.136700:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017649.136702:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017649.136703:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017649.136705:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1915 00000100:00080000:2.0:1734017649.136707:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017649.136709:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017649.136712:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017649.136717:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334318208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017649.136723:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017649.136724:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017649.136725:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017649.136726:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334317952:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017649.136727:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1915 00000100:00080000:2.0:1734017649.136729:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017649.136731:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017649.136733:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017649.136738:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334318336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017649.136747:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017649.136749:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017649.136750:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017649.136752:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1915 00000100:00080000:2.0:1734017649.136753:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017649.136755:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017649.136757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017649.136762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334318464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017649.136767:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017649.136768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017649.136769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017649.136771:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1915 00000800:00000100:1.0:1734017649.136773:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017649.136773:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017649.136775:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017649.136777:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017649.136782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334318592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017649.136786:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017649.136787:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017649/real 1734017649] req@ffff8800a78a1f80 x1818247334317952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017649.136788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017649.136789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017649.136791:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1915 00000100:00080000:2.0:1734017649.136793:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017649.136795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017649.136797:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017649.136802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334318720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017649.136805:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334317952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017649.136810:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017649.136812:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017649.136815:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334317952:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017649.136819:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334318080:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017649.136829:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017649.136833:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017649/real 1734017649] req@ffff8800a78a0a80 x1818247334318080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017649.136840:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334318080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017649.136844:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017649.136846:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017649.136847:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334318080:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017649.136862:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334318208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017649.136872:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017649.136875:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017649/real 1734017649] req@ffff8800a78a2680 x1818247334318208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017649.136881:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334318208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017649.136885:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017649.136887:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017649.136889:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334318208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017649.136892:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334318336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017649.136900:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017649.136903:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017649/real 1734017649] req@ffff8800a78a0700 x1818247334318336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017649.136909:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334318336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017649.136914:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017649.136915:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017649.136917:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334318336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017649.136920:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334318464:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017649.136928:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017649.136931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017649/real 1734017649] req@ffff8800a78a1880 x1818247334318464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017649.136937:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334318464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017649.136941:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017649.136942:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017649.136944:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334318464:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017649.136948:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334318592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017649.136957:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017649.136961:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017649/real 1734017649] req@ffff8800a78a3b80 x1818247334318592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017649.136967:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334318592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017649.136971:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017649.136972:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017649.136974:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334318592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017649.136977:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334318720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017649.136984:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017649.136988:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017649/real 1734017649] req@ffff8800a78a3480 x1818247334318720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017649.136993:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334318720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017649.136997:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017649.136998:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017649.137000:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334318720:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017649.644752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017650.646754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017651.648747:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017652.650726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017653.652737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017654.144615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017654.144624:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017654.144629:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017654.144639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1920 00000100:00080000:2.0:1734017654.144647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017654.144655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017654.144694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334318848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017654.144729:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017654.144733:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017654.144735:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017654.144740:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1920 00000100:00080000:2.0:1734017654.144743:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017654.144747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017654.144757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017654.144768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334318976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017654.144778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017654.144781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017654.144784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017654.144788:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1920 00000100:00080000:2.0:1734017654.144791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017654.144794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017654.144799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017654.144808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334319104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017654.144817:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017654.144820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017654.144821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017654.144825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1920 00000100:00080000:2.0:1734017654.144828:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017654.144831:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017654.144835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017654.144838:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334318848:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017654.144844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334319232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017654.144853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017654.144856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017654.144858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017654.144861:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1920 00000100:00080000:2.0:1734017654.144864:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017654.144867:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017654.144871:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017654.144878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334319360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017654.144887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017654.144890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017654.144892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017654.144895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1920 00000100:00080000:2.0:1734017654.144898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017654.144902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017654.144905:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017654.144905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017654.144913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334319488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017654.144921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017654.144924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017654.144925:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017654.144927:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017654/real 1734017654] req@ffff8800a78a3480 x1818247334318848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017654.144929:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1920 00000100:00080000:2.0:1734017654.144931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017654.144934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017654.144938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017654.144945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334319616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017654.144955:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334318848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017654.144965:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017654.144968:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017654.144973:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334318848:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017654.144980:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334318976:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017654.144997:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017654.145004:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017654/real 1734017654] req@ffff8800a78a3b80 x1818247334318976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017654.145014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334318976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017654.145022:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017654.145025:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017654.145028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334318976:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017654.145034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334319104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017654.145048:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017654.145054:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017654/real 1734017654] req@ffff8800a78a1880 x1818247334319104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017654.145064:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334319104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017654.145071:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017654.145074:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017654.145077:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334319104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017654.145101:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334319232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017654.145114:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017654.145120:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017654/real 1734017654] req@ffff8800a78a0700 x1818247334319232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017654.145131:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334319232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017654.145138:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017654.145140:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017654.145143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334319232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017654.145149:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334319360:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017654.145161:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017654.145167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017654/real 1734017654] req@ffff8800a78a2680 x1818247334319360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017654.145177:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334319360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017654.145184:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017654.145186:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017654.145189:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334319360:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017654.145195:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334319488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017654.145210:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017654.145216:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017654/real 1734017654] req@ffff8800a78a0a80 x1818247334319488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017654.145226:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334319488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017654.145233:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017654.145235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017654.145238:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334319488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017654.145243:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334319616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017654.145255:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017654.145261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017654/real 1734017654] req@ffff8800a78a1f80 x1818247334319616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017654.145271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334319616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017654.145277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017654.145280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017654.145283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334319616:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017654.654724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017655.656730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017656.658716:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017657.660747:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017658.662748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017659.152594:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017659.152603:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017659.152608:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017659.152618:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1925 00000100:00080000:2.0:1734017659.152626:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017659.152634:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017659.152672:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334319744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017659.152708:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017659.152712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017659.152714:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017659.152719:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1925 00000100:00080000:2.0:1734017659.152722:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017659.152726:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017659.152736:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017659.152747:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334319872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017659.152757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017659.152761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017659.152762:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017659.152767:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1925 00000100:00080000:2.0:1734017659.152769:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017659.152773:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017659.152777:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017659.152786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334320000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017659.152795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017659.152798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017659.152800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017659.152804:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1925 00000100:00080000:2.0:1734017659.152806:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017659.152809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017659.152813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017659.152821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334320128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017659.152824:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334319744:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017659.152830:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017659.152833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017659.152834:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017659.152838:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1925 00000100:00080000:2.0:1734017659.152840:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017659.152843:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017659.152847:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017659.152855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334320256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017659.152862:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017659.152865:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017659.152867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017659.152870:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1925 00000100:00080000:2.0:1734017659.152882:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017659.152886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017659.152888:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017659.152891:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017659.152899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334320384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017659.152908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017659.152909:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017659/real 1734017659] req@ffff8800a78a1f80 x1818247334319744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017659.152911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017659.152913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017659.152916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1925 00000100:00080000:2.0:1734017659.152919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017659.152922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017659.152926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017659.152933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334320512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017659.152937:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334319744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017659.152947:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017659.152951:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017659.152955:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334319744:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017659.152963:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334319872:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017659.152979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017659.152986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017659/real 1734017659] req@ffff8800a78a0a80 x1818247334319872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017659.152997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334319872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017659.153005:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017659.153007:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017659.153011:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334319872:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017659.153016:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334320000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017659.153030:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017659.153036:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017659/real 1734017659] req@ffff8800a78a2680 x1818247334320000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017659.153046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334320000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017659.153053:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017659.153056:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017659.153059:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334320000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017659.153083:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334320128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017659.153096:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017659.153102:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017659/real 1734017659] req@ffff8800a78a0700 x1818247334320128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017659.153113:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334320128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017659.153120:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017659.153122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017659.153125:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334320128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017659.153130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334320256:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017659.153143:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017659.153148:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017659/real 1734017659] req@ffff8800a78a1880 x1818247334320256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017659.153158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334320256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017659.153165:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017659.153167:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017659.153170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334320256:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017659.153176:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334320384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017659.153191:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017659.153197:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017659/real 1734017659] req@ffff8800a78a3b80 x1818247334320384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017659.153208:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334320384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017659.153215:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017659.153217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017659.153220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334320384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017659.153225:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334320512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017659.153237:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017659.153243:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017659/real 1734017659] req@ffff8800a78a3480 x1818247334320512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017659.153253:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334320512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017659.153260:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017659.153262:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017659.153265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334320512:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017659.664747:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017660.666729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017661.668755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017662.670743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017663.672738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017664.160614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017664.160626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017664.160632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017664.160645:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1930 00000100:00080000:2.0:1734017664.160656:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017664.160666:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017664.160716:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334320640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017664.160756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017664.160760:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017664.160762:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017664.160767:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1930 00000100:00080000:2.0:1734017664.160770:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017664.160774:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017664.160786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017664.160799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334320768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017664.160809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017664.160812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017664.160814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017664.160818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1930 00000100:00080000:2.0:1734017664.160821:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017664.160825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017664.160829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017664.160839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334320896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017664.160848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017664.160851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017664.160852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017664.160856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1930 00000100:00080000:2.0:1734017664.160859:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017664.160862:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017664.160867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017664.160876:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334320640:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017664.160876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334321024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017664.160886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017664.160889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017664.160891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017664.160895:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1930 00000100:00080000:2.0:1734017664.160898:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017664.160901:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017664.160906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017664.160915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334321152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017664.160923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017664.160926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017664.160927:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017664.160931:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1930 00000100:00080000:2.0:1734017664.160934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017664.160938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017664.160942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017664.160950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334321280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017664.160957:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017664.160961:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017664.160962:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017664.160966:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1930 00000100:00080000:2.0:1734017664.160969:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017664.160972:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017664.160975:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334320768:192.168.201.116@tcp:38:kworker.0 00000008:00100000:2.0:1734017664.160977:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017664.160985:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334321408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017664.160996:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334320896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017664.161020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334321024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017664.161037:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334321152:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017664.161054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334321280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017664.161069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334321408:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017664.674736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017665.676725:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017666.678755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017667.166750:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:0.0:1734017667.166796:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017667.166814:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017667.166847:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017667.166855:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017667.166863:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017667.166869:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017667.166875:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017667.166881:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734017667.166951:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017664/real 1734017667] req@ffff8800a78a0700 x1818247334321024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017667.166991:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334321024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017667.167007:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017667.167012:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017667.167019:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334321024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017667.167031:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017664/real 1734017667] req@ffff8800a78a2680 x1818247334321152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017667.167051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334321152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017667.167063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017667.167067:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017667.167073:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334321152:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017667.167084:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017664/real 1734017667] req@ffff8800a78a0a80 x1818247334321280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017667.167102:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334321280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017667.167115:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017667.167119:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017667.167125:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334321280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017667.167134:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017664/real 1734017667] req@ffff8800a78a1f80 x1818247334321408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017667.167152:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334321408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017667.167164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017667.167168:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017667.167173:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334321408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017667.167182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017664/real 1734017667] req@ffff8800a78a3480 x1818247334320640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017667.167267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334320640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017667.167280:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017667.167285:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017667.167290:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334320640:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017667.167300:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017664/real 1734017667] req@ffff8800a78a3b80 x1818247334320768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017667.167320:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334320768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017667.167342:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017667.167346:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017667.167352:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334320768:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017667.167362:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017664/real 1734017667] req@ffff8800a78a1880 x1818247334320896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017667.167380:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334320896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017667.167392:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017667.167397:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017667.167402:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334320896:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017667.680722:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017668.682742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017669.168603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017669.168612:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017669.168617:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017669.168627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1935 00000100:00080000:2.0:1734017669.168635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017669.168643:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017669.168682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334321536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017669.168719:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017669.168725:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017669.168728:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017669.168735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1935 00000100:00080000:2.0:1734017669.168739:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017669.168745:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017669.168759:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017669.168778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334321664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017669.168786:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334321536:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017669.168794:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017669.168798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017669.168801:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017669.168807:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1935 00000100:00080000:2.0:1734017669.168811:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017669.168816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017669.168823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017669.168835:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334321792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017669.168847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017669.168851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017669.168854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017669.168855:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017669.168858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1935 00000100:00080000:2.0:1734017669.168863:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017669.168867:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017669.168873:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017669.168877:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017669/real 1734017669] req@ffff8800a78a1880 x1818247334321536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017669.168885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334321920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017669.168898:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017669.168899:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334321536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017669.168903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017669.168905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017669.168910:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1935 00000100:00080000:1.0:1734017669.168914:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017669.168915:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734017669.168918:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734017669.168920:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017669.168924:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334321536:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017669.168925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017669.168937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334322048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017669.168949:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017669.168951:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334321664:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017669.168954:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017669.168957:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017669.168962:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1935 00000100:00080000:2.0:1734017669.168966:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017669.168972:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017669.168976:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017669.168978:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017669.168988:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017669/real 1734017669] req@ffff8800a78a3b80 x1818247334321664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017669.168990:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334322176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017669.169004:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017669.169005:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334321664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017669.169009:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017669.169012:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017669.169017:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1935 00000100:00080000:1.0:1734017669.169018:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017669.169021:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734017669.169022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734017669.169026:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017669.169027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334321664:192.168.201.116@tcp:38:kworker.0 00000008:00100000:2.0:1734017669.169032:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017669.169036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334321792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734017669.169043:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334322304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017669.169058:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017669.169069:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017669/real 1734017669] req@ffff8800a78a3480 x1818247334321792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017669.169086:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334321792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017669.169098:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017669.169102:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017669.169107:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334321792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017669.169117:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334321920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017669.169142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017669.169153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017669/real 1734017669] req@ffff8800a78a1f80 x1818247334321920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017669.169209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334321920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017669.169225:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017669.169228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017669.169231:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334321920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017669.169249:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334322048:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017669.169268:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017669.169275:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017669/real 1734017669] req@ffff8800a78a0a80 x1818247334322048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017669.169285:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334322048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017669.169292:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017669.169295:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017669.169298:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334322048:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017669.169303:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334322176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017669.169315:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017669.169321:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017669/real 1734017669] req@ffff8800a78a2680 x1818247334322176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017669.169331:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334322176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017669.169338:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017669.169340:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017669.169343:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334322176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017669.169349:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334322304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017669.169361:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017669.169367:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017669/real 1734017669] req@ffff8800a78a0700 x1818247334322304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017669.169377:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334322304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017669.169384:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017669.169386:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017669.169389:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334322304:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017669.684760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017670.686728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017671.688759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017672.690753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017673.692732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017674.176617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017674.176631:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017674.176636:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017674.176646:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1940 00000100:00080000:2.0:1734017674.176655:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017674.176662:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017674.176701:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334322432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017674.176742:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017674.176746:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017674.176748:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017674.176752:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1940 00000100:00080000:2.0:1734017674.176755:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017674.176759:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017674.176770:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017674.176781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334322560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017674.176791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017674.176794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017674.176796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017674.176800:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1940 00000100:00080000:2.0:1734017674.176802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017674.176806:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017674.176811:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017674.176820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334322688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017674.176828:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017674.176831:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017674.176833:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017674.176836:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1940 00000100:00080000:2.0:1734017674.176839:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017674.176843:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017674.176847:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017674.176855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334322816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017674.176856:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334322432:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017674.176864:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017674.176867:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017674.176868:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017674.176872:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1940 00000100:00080000:2.0:1734017674.176874:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017674.176877:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017674.176881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017674.176888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334322944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017674.176896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017674.176899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017674.176900:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017674.176904:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1940 00000100:00080000:2.0:1734017674.176906:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017674.176909:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017674.176914:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017674.176921:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334323072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017674.176923:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017674.176929:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017674.176933:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017674.176935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017674.176938:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1940 00000100:00080000:2.0:1734017674.176941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017674.176944:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017674/real 1734017674] req@ffff8800a78a0700 x1818247334322432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017674.176944:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017674.176948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017674.176955:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334323200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017674.176964:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334322432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017674.176974:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017674.176978:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017674.176983:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334322432:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017674.176990:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334322560:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017674.177007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017674.177014:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017674/real 1734017674] req@ffff8800a78a2680 x1818247334322560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017674.177033:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334322560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017674.177041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017674.177048:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017674.177052:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334322560:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017674.177057:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334322688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017674.177103:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017674.177112:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017674/real 1734017674] req@ffff8800a78a0a80 x1818247334322688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017674.177123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334322688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017674.177130:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017674.177133:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017674.177136:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334322688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017674.177161:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334322816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017674.177177:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017674.177183:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017674/real 1734017674] req@ffff8800a78a1f80 x1818247334322816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017674.177194:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334322816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017674.177201:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017674.177204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017674.177208:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334322816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017674.177214:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334322944:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017674.177226:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017674.177232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017674/real 1734017674] req@ffff8800a78a3480 x1818247334322944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017674.177243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334322944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017674.177251:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017674.177253:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017674.177256:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334322944:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017674.177261:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334323072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017674.177273:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017674.177279:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017674/real 1734017674] req@ffff8800a78a3b80 x1818247334323072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017674.177289:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334323072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017674.177296:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017674.177298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017674.177301:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334323072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017674.177306:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334323200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017674.177321:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017674.177327:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017674/real 1734017674] req@ffff8800a78a1880 x1818247334323200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017674.177336:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334323200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017674.177343:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017674.177345:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017674.177348:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334323200:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017674.694764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017675.696739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017676.698750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017677.700755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017678.702759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017679.184610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017679.184619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017679.184623:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017679.184634:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1945 00000100:00080000:2.0:1734017679.184642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017679.184650:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017679.184687:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334323328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017679.184723:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017679.184727:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017679.184730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017679.184734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1945 00000100:00080000:2.0:1734017679.184737:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017679.184741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017679.184752:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017679.184763:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334323456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017679.184774:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017679.184777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017679.184779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017679.184783:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1945 00000100:00080000:2.0:1734017679.184785:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017679.184789:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017679.184793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017679.184802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334323584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017679.184810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017679.184813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017679.184815:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017679.184818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1945 00000100:00080000:2.0:1734017679.184821:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017679.184824:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017679.184827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017679.184836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334323712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017679.184840:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334323328:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017679.184845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017679.184848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017679.184850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017679.184853:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1945 00000100:00080000:2.0:1734017679.184856:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017679.184859:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017679.184863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017679.184870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334323840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017679.184878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017679.184881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017679.184882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017679.184885:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1945 00000100:00080000:2.0:1734017679.184888:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017679.184891:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017679.184895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017679.184902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334323968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017679.184909:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017679.184910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017679.184913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017679.184914:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017679.184918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1945 00000100:00080000:2.0:1734017679.184920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017679.184923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017679.184927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017679.184931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017679/real 1734017679] req@ffff8800a78a1880 x1818247334323328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017679.184934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334324096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017679.184951:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334323328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017679.184961:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017679.184965:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017679.184969:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334323328:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017679.184977:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334323456:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017679.185035:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017679.185043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017679/real 1734017679] req@ffff8800a78a3b80 x1818247334323456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017679.185055:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334323456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017679.185063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017679.185066:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017679.185070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334323456:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017679.185076:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334323584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017679.185090:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017679.185096:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017679/real 1734017679] req@ffff8800a78a3480 x1818247334323584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017679.185107:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334323584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017679.185114:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017679.185117:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017679.185120:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334323584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017679.185144:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334323712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017679.185157:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017679.185163:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017679/real 1734017679] req@ffff8800a78a1f80 x1818247334323712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017679.185173:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334323712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017679.185181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017679.185183:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017679.185187:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334323712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017679.185193:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334323840:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017679.185206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017679.185212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017679/real 1734017679] req@ffff8800a78a0a80 x1818247334323840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017679.185222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334323840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017679.185229:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017679.185231:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017679.185234:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334323840:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017679.185239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334323968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017679.185251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017679.185257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017679/real 1734017679] req@ffff8800a78a2680 x1818247334323968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017679.185267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334323968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017679.185275:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017679.185277:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017679.185280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334323968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017679.185285:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334324096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017679.185297:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017679.185303:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017679/real 1734017679] req@ffff8800a78a0700 x1818247334324096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017679.185313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334324096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017679.185320:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017679.185323:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017679.185326:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334324096:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017679.704758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017680.706737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017681.708766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017682.710703:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017683.712767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017684.192621:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017684.192635:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017684.192640:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017684.192654:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1950 00000100:00080000:2.0:1734017684.192665:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017684.192674:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017684.192725:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334324224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017684.192765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017684.192769:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017684.192771:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017684.192776:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1950 00000100:00080000:2.0:1734017684.192779:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017684.192784:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017684.192795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017684.192807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334324352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017684.192818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017684.192821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017684.192823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017684.192827:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1950 00000100:00080000:2.0:1734017684.192830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017684.192834:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017684.192839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017684.192849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334324480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017684.192859:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017684.192862:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017684.192864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017684.192867:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1950 00000100:00080000:2.0:1734017684.192870:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017684.192873:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017684.192877:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017684.192886:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334324608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017684.192895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017684.192898:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017684.192933:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017684.192937:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1950 00000100:00080000:2.0:1734017684.192940:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017684.192944:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017684.192946:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334324224:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017684.192949:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017684.192959:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334324736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017684.192969:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017684.192972:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017684.192974:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017684.192978:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1950 00000100:00080000:2.0:1734017684.192981:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017684.192985:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017684.192989:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017684.192998:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334324864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017684.193006:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017684.193009:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017684.193011:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017684.193014:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1950 00000100:00080000:2.0:1734017684.193017:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017684.193019:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017684.193021:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017684.193025:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017684.193033:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334324992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017684.193043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017684/real 1734017684] req@ffff8800a78a0700 x1818247334324224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017684.193063:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334324224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017684.193075:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017684.193079:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017684.193084:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334324224:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017684.193092:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334324352:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017684.193109:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017684.193116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017684/real 1734017684] req@ffff8800a78a2680 x1818247334324352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017684.193127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334324352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017684.193135:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017684.193138:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017684.193142:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334324352:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017684.193149:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334324480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017684.193163:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017684.193169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017684/real 1734017684] req@ffff8800a78a0a80 x1818247334324480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017684.193179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334324480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017684.193187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017684.193189:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017684.193192:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334324480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017684.193218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334324608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017684.193232:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017684.193238:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017684/real 1734017684] req@ffff8800a78a1f80 x1818247334324608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017684.193248:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334324608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017684.193256:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017684.193259:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017684.193262:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334324608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017684.193268:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334324736:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017684.193284:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017684.193291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017684/real 1734017684] req@ffff8800a78a3480 x1818247334324736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017684.193302:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334324736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017684.193309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017684.193312:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017684.193315:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334324736:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017684.193321:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334324864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017684.193333:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017684.193339:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017684/real 1734017684] req@ffff8800a78a3b80 x1818247334324864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017684.193349:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334324864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017684.193356:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017684.193359:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017684.193362:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334324864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017684.193367:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334324992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017684.193380:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017684.193386:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017684/real 1734017684] req@ffff8800a78a1880 x1818247334324992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017684.193402:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334324992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017684.193409:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017684.193411:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017684.193414:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334324992:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017684.714743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017685.716696:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017686.718709:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017687.720701:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017688.722740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017689.200617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017689.200626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017689.200631:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017689.200641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1955 00000100:00080000:2.0:1734017689.200650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017689.200657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017689.200695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334325120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017689.200737:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017689.200741:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017689.200742:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017689.200747:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1955 00000100:00080000:2.0:1734017689.200750:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017689.200754:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017689.200764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017689.200775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334325248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017689.200785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017689.200788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017689.200789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017689.200794:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1955 00000100:00080000:2.0:1734017689.200797:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017689.200800:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017689.200805:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017689.200844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334325376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017689.200854:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017689.200857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017689.200859:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017689.200863:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1955 00000100:00080000:2.0:1734017689.200866:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017689.200870:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017689.200874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017689.200880:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334325120:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017689.200884:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334325504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017689.200893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017689.200896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017689.200898:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017689.200901:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1955 00000100:00080000:2.0:1734017689.200904:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017689.200907:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017689.200911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017689.200918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334325632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017689.200926:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017689.200929:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017689.200930:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017689.200934:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1955 00000100:00080000:2.0:1734017689.200936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017689.200940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017689.200944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017689.200945:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017689.200951:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334325760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017689.200959:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017689.200962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017689.200964:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017689.200966:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017689/real 1734017689] req@ffff8800a78a1880 x1818247334325120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017689.200968:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1955 00000100:00080000:2.0:1734017689.200970:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017689.200974:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017689.200977:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017689.200984:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334325888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017689.200986:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334325120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017689.200997:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017689.201001:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017689.201005:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334325120:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017689.201012:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334325248:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017689.201029:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017689.201036:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017689/real 1734017689] req@ffff8800a78a3b80 x1818247334325248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017689.201047:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334325248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017689.201055:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017689.201058:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017689.201061:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334325248:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017689.201067:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334325376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017689.201081:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017689.201087:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017689/real 1734017689] req@ffff8800a78a3480 x1818247334325376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017689.201097:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334325376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017689.201105:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017689.201107:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017689.201110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334325376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017689.201138:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334325504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017689.201151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017689.201157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017689/real 1734017689] req@ffff8800a78a1f80 x1818247334325504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017689.201167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334325504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017689.201175:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017689.201177:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017689.201180:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334325504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017689.201186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334325632:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017689.201201:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017689.201207:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017689/real 1734017689] req@ffff8800a78a0a80 x1818247334325632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017689.201218:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334325632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017689.201225:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017689.201227:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017689.201230:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334325632:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017689.201235:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334325760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017689.201247:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017689.201253:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017689/real 1734017689] req@ffff8800a78a2680 x1818247334325760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017689.201262:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334325760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017689.201269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017689.201272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017689.201274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334325760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017689.201280:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334325888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017689.201292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017689.201298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017689/real 1734017689] req@ffff8800a78a0700 x1818247334325888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017689.201315:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334325888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017689.201322:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017689.201325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017689.201327:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334325888:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017689.724768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017690.726736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017691.728749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017692.730720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017693.732803:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017694.208586:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017694.208596:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017694.208600:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017694.208611:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1960 00000100:00080000:2.0:1734017694.208619:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017694.208627:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017694.208665:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334326016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017694.208699:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017694.208703:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017694.208705:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017694.208709:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1960 00000100:00080000:2.0:1734017694.208713:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017694.208716:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017694.208726:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017694.208785:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334326144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017694.208796:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017694.208800:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017694.208802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017694.208806:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1960 00000100:00080000:2.0:1734017694.208809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017694.208812:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017694.208817:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017694.208827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334326272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017694.208835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017694.208838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017694.208840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017694.208844:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1960 00000100:00080000:2.0:1734017694.208846:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017694.208849:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017694.208852:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334326016:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017694.208853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017694.208861:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334326400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017694.208870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017694.208873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017694.208874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017694.208878:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1960 00000100:00080000:2.0:1734017694.208881:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017694.208884:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017694.208888:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017694.208895:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334326528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017694.208903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017694.208906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017694.208908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017694.208911:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1960 00000100:00080000:2.0:1734017694.208914:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017694.208917:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017694.208918:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017694.208921:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017694.208928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334326656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017694.208936:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017694.208939:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017694.208940:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017694/real 1734017694] req@ffff8800a78a0700 x1818247334326016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017694.208941:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017694.208944:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1960 00000100:00080000:2.0:1734017694.208947:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017694.208950:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017694.208954:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017694.208959:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334326016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017694.208961:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334326784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017694.208970:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017694.208973:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017694.208978:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334326016:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017694.208986:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334326144:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017694.209002:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017694.209009:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017694/real 1734017694] req@ffff8800a78a2680 x1818247334326144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017694.209020:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334326144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017694.209027:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017694.209031:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017694.209034:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334326144:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017694.209040:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334326272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017694.209054:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017694.209060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017694/real 1734017694] req@ffff8800a78a0a80 x1818247334326272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017694.209070:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334326272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017694.209077:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017694.209080:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017694.209083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334326272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017694.209106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334326400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017694.209118:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017694.209125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017694/real 1734017694] req@ffff8800a78a1f80 x1818247334326400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017694.209135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334326400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017694.209143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017694.209145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017694.209148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334326400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017694.209153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334326528:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017694.209169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017694.209175:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017694/real 1734017694] req@ffff8800a78a3480 x1818247334326528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017694.209185:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334326528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017694.209192:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017694.209195:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017694.209197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334326528:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017694.209203:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334326656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017694.209214:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017694.209220:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017694/real 1734017694] req@ffff8800a78a3b80 x1818247334326656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017694.209230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334326656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017694.209237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017694.209239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017694.209242:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334326656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017694.209248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334326784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017694.209260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017694.209266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017694/real 1734017694] req@ffff8800a78a1880 x1818247334326784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017694.209283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334326784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017694.209291:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017694.209293:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017694.209296:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334326784:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017694.734708:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017695.736767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017696.738747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017697.740763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017698.742756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017699.216576:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017699.216586:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017699.216590:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017699.216600:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1965 00000100:00080000:2.0:1734017699.216610:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017699.216618:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017699.216682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334326912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017699.216713:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017699.216717:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017699.216719:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017699.216724:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1965 00000100:00080000:2.0:1734017699.216728:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017699.216732:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017699.216743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017699.216755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334327040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017699.216765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017699.216768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017699.216770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017699.216774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1965 00000100:00080000:2.0:1734017699.216776:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017699.216780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017699.216786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017699.216795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334327168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017699.216804:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017699.216807:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017699.216808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017699.216811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1965 00000100:00080000:2.0:1734017699.216814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017699.216817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017699.216821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017699.216824:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334326912:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017699.216829:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334327296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017699.216838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017699.216841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017699.216843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017699.216846:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1965 00000100:00080000:2.0:1734017699.216848:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017699.216851:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017699.216855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017699.216863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334327424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017699.216872:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017699.216875:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017699.216876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017699.216880:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1965 00000100:00080000:2.0:1734017699.216883:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017699.216886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017699.216890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017699.216891:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017699.216897:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334327552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017699.216905:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017699.216908:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017699.216909:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017699.216913:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017699/real 1734017699] req@ffff8800a78a1880 x1818247334326912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017699.216913:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1965 00000100:00080000:2.0:1734017699.216915:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017699.216918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017699.216922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017699.216929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334327680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017699.216932:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334326912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017699.216942:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017699.216946:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017699.216951:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334326912:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017699.216958:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334327040:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017699.216975:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017699.216981:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017699/real 1734017699] req@ffff8800a78a3b80 x1818247334327040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017699.216992:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334327040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017699.217000:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017699.217003:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017699.217007:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334327040:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017699.217013:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334327168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017699.217027:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017699.217034:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017699/real 1734017699] req@ffff8800a78a3480 x1818247334327168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017699.217044:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334327168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017699.217051:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017699.217054:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017699.217057:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334327168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017699.217087:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334327296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017699.217099:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017699.217105:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017699/real 1734017699] req@ffff8800a78a1f80 x1818247334327296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017699.217116:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334327296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017699.217124:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017699.217126:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017699.217129:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334327296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017699.217135:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334327424:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017699.217150:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017699.217156:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017699/real 1734017699] req@ffff8800a78a0a80 x1818247334327424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017699.217167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334327424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017699.217174:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017699.217176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017699.217179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334327424:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017699.217185:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334327552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017699.217196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017699.217202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017699/real 1734017699] req@ffff8800a78a2680 x1818247334327552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017699.217212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334327552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017699.217219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017699.217221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017699.217224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334327552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017699.217229:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334327680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017699.217241:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017699.217247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017699/real 1734017699] req@ffff8800a78a0700 x1818247334327680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017699.217264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334327680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017699.217271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017699.217273:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017699.217276:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334327680:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017699.744811:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017700.746778:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017701.748802:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017702.750774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017703.752757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017704.224643:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017704.224652:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017704.224657:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017704.224668:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1970 00000100:00080000:2.0:1734017704.224676:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017704.224683:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017704.224720:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334327808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017704.224760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017704.224764:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017704.224765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017704.224770:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1970 00000100:00080000:2.0:1734017704.224773:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017704.224777:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017704.224787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017704.224799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334327936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017704.224808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017704.224811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017704.224812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017704.224816:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1970 00000100:00080000:2.0:1734017704.224819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017704.224823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017704.224827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017704.224836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334328064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017704.224844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017704.224848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017704.224849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017704.224852:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1970 00000100:00080000:2.0:1734017704.224855:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017704.224858:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017704.224862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017704.224868:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334327808:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017704.224871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334328192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017704.224880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017704.224883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017704.224885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017704.224888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1970 00000100:00080000:2.0:1734017704.224890:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017704.224893:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017704.224897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017704.224904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334328320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017704.224912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017704.224915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017704.224916:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017704.224920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1970 00000100:00080000:2.0:1734017704.224931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017704.224935:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017704.224936:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017704.224939:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017704.224947:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334328448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017704.224955:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017704.224958:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017704/real 1734017704] req@ffff8800a78a0700 x1818247334327808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017704.224959:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017704.224961:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017704.224964:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1970 00000100:00080000:2.0:1734017704.224967:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017704.224970:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017704.224974:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017704.224977:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334327808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017704.224982:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334328576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017704.224988:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017704.224991:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017704.224996:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334327808:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017704.225003:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334327936:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017704.225020:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017704.225027:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017704/real 1734017704] req@ffff8800a78a2680 x1818247334327936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017704.225038:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334327936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017704.225046:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017704.225049:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017704.225052:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334327936:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017704.225059:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334328064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017704.225072:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017704.225078:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017704/real 1734017704] req@ffff8800a78a0a80 x1818247334328064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017704.225088:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334328064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017704.225096:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017704.225098:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017704.225102:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334328064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017704.225125:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334328192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017704.225138:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017704.225144:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017704/real 1734017704] req@ffff8800a78a1f80 x1818247334328192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017704.225155:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334328192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017704.225162:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017704.225165:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017704.225168:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334328192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017704.225173:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334328320:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017704.225189:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017704.225195:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017704/real 1734017704] req@ffff8800a78a3480 x1818247334328320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017704.225205:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334328320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017704.225213:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017704.225215:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017704.225218:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334328320:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017704.225223:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334328448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017704.225234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017704.225240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017704/real 1734017704] req@ffff8800a78a3b80 x1818247334328448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017704.225250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334328448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017704.225257:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017704.225259:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017704.225262:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334328448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017704.225267:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334328576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017704.225279:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017704.225285:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017704/real 1734017704] req@ffff8800a78a1880 x1818247334328576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017704.225302:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334328576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017704.225309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017704.225311:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017704.225314:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334328576:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017704.754770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017705.756743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017706.758740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017707.760765:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017708.762747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017709.232610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017709.232619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017709.232623:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017709.232634:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1975 00000100:00080000:2.0:1734017709.232642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017709.232650:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017709.232688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334328704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017709.232724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017709.232730:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017709.232732:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017709.232739:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1975 00000100:00080000:2.0:1734017709.232743:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017709.232749:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017709.232762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017709.232781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334328832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017709.232789:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334328704:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017709.232798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017709.232802:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017709.232805:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017709.232810:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1975 00000100:00080000:2.0:1734017709.232814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017709.232819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017709.232825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017709.232838:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334328960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017709.232850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017709.232855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017709.232857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017709.232860:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017709.232862:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1975 00000100:00080000:2.0:1734017709.232866:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017709.232870:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017709.232875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017709.232884:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017709/real 1734017709] req@ffff8800a78a1880 x1818247334328704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017709.232887:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334329088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017709.232901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017709.232905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017709.232907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017709.232909:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334328704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017709.232913:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1975 00000100:00080000:2.0:1734017709.232918:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734017709.232922:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017709.232923:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734017709.232927:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000008:00100000:2.0:1734017709.232929:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734017709.232933:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334328704:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017709.232940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334329216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017709.232951:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017709.232955:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1734017709.232958:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334328832:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017709.232958:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017709.232963:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1975 00000100:00080000:2.0:1734017709.232967:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017709.232972:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017709.232978:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017709.232983:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017709.232991:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334329344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017709.232995:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017709/real 1734017709] req@ffff8800a78a3b80 x1818247334328832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017709.233005:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017709.233010:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017709.233012:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334328832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017709.233013:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017709.233018:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1975 00000100:00080000:2.0:1734017709.233022:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734017709.233025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017709.233027:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734017709.233029:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000008:00100000:2.0:1734017709.233033:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017709.233034:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334328832:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017709.233044:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334328960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734017709.233044:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334329472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017709.233066:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017709.233076:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017709/real 1734017709] req@ffff8800a78a3480 x1818247334328960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017709.233093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334328960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017709.233105:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017709.233109:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017709.233114:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334328960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017709.233124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334329088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017709.233151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017709.233161:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017709/real 1734017709] req@ffff8800a78a1f80 x1818247334329088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017709.233172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334329088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017709.233180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017709.233182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017709.233185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334329088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017709.233191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334329216:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017709.233209:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017709.233215:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017709/real 1734017709] req@ffff8800a78a0a80 x1818247334329216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017709.233226:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334329216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017709.233233:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017709.233235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017709.233238:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334329216:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017709.233255:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334329344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017709.233267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017709.233274:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017709/real 1734017709] req@ffff8800a78a2680 x1818247334329344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017709.233283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334329344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017709.233290:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017709.233292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017709.233296:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334329344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017709.233301:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334329472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017709.233313:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017709.233319:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017709/real 1734017709] req@ffff8800a78a0700 x1818247334329472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017709.233335:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334329472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017709.233343:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017709.233345:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017709.233348:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334329472:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017709.764751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017710.766774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017711.768745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017712.770775:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017713.772787:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017714.240614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017714.240623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017714.240627:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017714.240638:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1980 00000100:00080000:2.0:1734017714.240646:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017714.240654:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017714.240693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334329600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017714.240730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017714.240734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017714.240736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017714.240741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1980 00000100:00080000:2.0:1734017714.240744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017714.240748:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017714.240758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017714.240770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334329728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017714.240780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017714.240783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017714.240784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017714.240789:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1980 00000100:00080000:2.0:1734017714.240791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017714.240795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017714.240800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017714.240809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334329856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017714.240818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017714.240821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017714.240823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017714.240826:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1980 00000100:00080000:2.0:1734017714.240828:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017714.240832:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017714.240836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017714.240843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334329984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017714.240852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017714.240856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017714.240857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017714.240860:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1980 00000100:00080000:2.0:1734017714.240863:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017714.240866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017714.240870:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334329600:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017714.240870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017714.240878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334330112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017714.240886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017714.240889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017714.240891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017714.240895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1980 00000100:00080000:2.0:1734017714.240898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017714.240901:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017714.240905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017714.240913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334330240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017714.240920:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017714.240923:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017714.240925:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017714.240928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1980 00000100:00080000:2.0:1734017714.240930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017714.240933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017714.240938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017714.240945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334330368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017714.240966:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017714.240994:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017714/real 1734017714] req@ffff8800a78a0700 x1818247334329600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017714.241015:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334329600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017714.241028:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017714.241032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017714.241038:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334329600:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017714.241046:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334329728:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017714.241065:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017714.241072:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017714/real 1734017714] req@ffff8800a78a2680 x1818247334329728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017714.241083:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334329728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017714.241092:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017714.241096:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017714.241099:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334329728:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017714.241106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334329856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017714.241122:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017714.241128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017714/real 1734017714] req@ffff8800a78a0a80 x1818247334329856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017714.241138:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334329856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017714.241146:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017714.241149:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017714.241153:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334329856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017714.241182:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334329984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017714.241197:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017714.241203:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017714/real 1734017714] req@ffff8800a78a1f80 x1818247334329984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017714.241214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334329984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017714.241222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017714.241225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017714.241228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334329984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017714.241234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334330112:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017714.241251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017714.241258:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017714/real 1734017714] req@ffff8800a78a3480 x1818247334330112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017714.241269:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334330112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017714.241276:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017714.241279:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017714.241281:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334330112:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017714.241287:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334330240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017714.241300:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017714.241306:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017714/real 1734017714] req@ffff8800a78a3b80 x1818247334330240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017714.241315:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334330240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017714.241323:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017714.241325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017714.241328:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334330240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017714.241334:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334330368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017714.241347:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017714.241353:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017714/real 1734017714] req@ffff8800a78a1880 x1818247334330368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017714.241371:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334330368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017714.241379:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017714.241381:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017714.241384:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334330368:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017714.774770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017715.776771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017716.778786:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017717.780748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017718.782735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017719.248600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017719.248610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017719.248615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017719.248625:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1985 00000100:00080000:2.0:1734017719.248634:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017719.248642:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017719.248681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334330496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017719.248724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017719.248728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017719.248730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017719.248734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1985 00000100:00080000:2.0:1734017719.248738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017719.248742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017719.248752:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017719.248764:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334330624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017719.248773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017719.248776:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017719.248778:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017719.248782:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1985 00000100:00080000:2.0:1734017719.248785:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017719.248789:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017719.248793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017719.248802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334330752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017719.248811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017719.248814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017719.248816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017719.248819:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1985 00000100:00080000:2.0:1734017719.248822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017719.248826:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017719.248830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017719.248837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334330880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017719.248847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017719.248850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017719.248851:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017719.248853:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334330496:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017719.248855:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1985 00000100:00080000:2.0:1734017719.248857:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017719.248861:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017719.248865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017719.248873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334331008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017719.248881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017719.248884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017719.248885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017719.248889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1985 00000100:00080000:2.0:1734017719.248891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017719.248894:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017719.248898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017719.248906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334331136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017719.248914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017719.248918:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017719.248919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017719.248923:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1985 00000100:00080000:2.0:1734017719.248926:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017719.248929:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017719.248933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017719.248940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334331264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017719.248947:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017719.248974:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017719/real 1734017719] req@ffff8800a78a1880 x1818247334330496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017719.248995:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334330496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017719.249008:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017719.249012:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017719.249018:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334330496:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017719.249025:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334330624:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017719.249043:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017719.249050:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017719/real 1734017719] req@ffff8800a78a3b80 x1818247334330624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017719.249062:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334330624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017719.249070:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017719.249074:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017719.249078:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334330624:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017719.249085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334330752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017719.249100:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017719.249106:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017719/real 1734017719] req@ffff8800a78a3480 x1818247334330752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017719.249117:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334330752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017719.249124:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017719.249127:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017719.249130:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334330752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017719.249159:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334330880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017719.249172:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017719.249178:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017719/real 1734017719] req@ffff8800a78a1f80 x1818247334330880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017719.249189:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334330880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017719.249197:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017719.249200:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017719.249203:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334330880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017719.249209:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334331008:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017719.249226:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017719.249233:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017719/real 1734017719] req@ffff8800a78a0a80 x1818247334331008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017719.249244:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334331008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017719.249251:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017719.249254:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017719.249257:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334331008:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017719.249262:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334331136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017719.249274:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017719.249280:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017719/real 1734017719] req@ffff8800a78a2680 x1818247334331136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017719.249290:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334331136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017719.249297:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017719.249300:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017719.249345:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334331136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017719.249354:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334331264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017719.249372:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017719.249380:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017719/real 1734017719] req@ffff8800a78a0700 x1818247334331264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017719.249398:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334331264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017719.249406:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017719.249409:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017719.249412:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334331264:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017719.784773:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017720.786742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017721.788719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017722.790735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017723.792741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017724.256604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017724.256613:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017724.256618:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017724.256628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1990 00000100:00080000:2.0:1734017724.256637:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017724.256645:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017724.256682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334331392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017724.256718:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017724.256722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017724.256724:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017724.256729:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1990 00000100:00080000:2.0:1734017724.256732:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017724.256736:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017724.256746:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017724.256757:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334331520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017724.256769:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017724.256772:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017724.256774:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017724.256778:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1990 00000100:00080000:2.0:1734017724.256780:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017724.256784:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017724.256788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017724.256797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334331648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017724.256805:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017724.256808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017724.256810:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017724.256813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1990 00000100:00080000:2.0:1734017724.256816:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017724.256819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017724.256823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017724.256830:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334331392:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017724.256831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334331776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017724.256841:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017724.256844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017724.256845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017724.256849:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1990 00000100:00080000:2.0:1734017724.256851:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017724.256855:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017724.256859:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017724.256867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334331904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017724.256875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017724.256878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017724.256879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017724.256882:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1990 00000100:00080000:2.0:1734017724.256886:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017724.256889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017724.256893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017724.256897:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017724.256901:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334332032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017724.256909:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017724.256912:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017724.256913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017724.256917:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1990 00000100:00100000:1.0:1734017724.256918:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017724/real 1734017724] req@ffff8800a78a0700 x1818247334331392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017724.256920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017724.256923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017724.256927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017724.256934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334332160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017724.256938:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334331392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017724.256949:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017724.256952:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017724.256957:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334331392:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017724.256964:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334331520:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017724.256981:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017724.256988:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017724/real 1734017724] req@ffff8800a78a2680 x1818247334331520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017724.256999:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334331520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017724.257007:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017724.257010:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017724.257014:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334331520:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017724.257020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334331648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017724.257034:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017724.257040:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017724/real 1734017724] req@ffff8800a78a0a80 x1818247334331648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017724.257050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334331648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017724.257057:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017724.257060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017724.257064:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334331648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017724.257088:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334331776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017724.257101:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017724.257107:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017724/real 1734017724] req@ffff8800a78a1f80 x1818247334331776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017724.257117:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334331776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017724.257125:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017724.257127:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017724.257130:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334331776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017724.257136:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334331904:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017724.257151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017724.257157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017724/real 1734017724] req@ffff8800a78a3480 x1818247334331904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017724.257167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334331904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017724.257175:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017724.257177:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017724.257180:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334331904:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017724.257186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334332032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017724.257197:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017724.257203:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017724/real 1734017724] req@ffff8800a78a3b80 x1818247334332032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017724.257213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334332032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017724.257264:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017724.257266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017724.257270:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334332032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017724.257276:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334332160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017724.257293:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017724.257301:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017724/real 1734017724] req@ffff8800a78a1880 x1818247334332160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017724.257318:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334332160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017724.257325:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017724.257327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017724.257330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334332160:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017724.794730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017725.796736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017726.798740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017727.800757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017728.802731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017729.264699:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017729.264708:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017729.264713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017729.264723:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 1995 00000100:00080000:0.0:1734017729.264732:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017729.264739:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017729.264775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334332288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017729.264820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017729.264824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017729.264826:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017729.264831:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1995 00000100:00080000:0.0:1734017729.264834:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017729.264838:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017729.264848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017729.264860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334332416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017729.264869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017729.264872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017729.264874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017729.264878:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1995 00000100:00080000:0.0:1734017729.264881:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017729.264884:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017729.264889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017729.264899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334332544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017729.264907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017729.264910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017729.264912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017729.264915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 1995 00000100:00080000:0.0:1734017729.264918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017729.264921:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017729.264925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017729.264934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334332672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017729.264934:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334332288:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734017729.264942:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017729.264945:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017729.264947:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017729.264950:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1995 00000100:00080000:0.0:1734017729.264953:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017729.264956:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017729.264960:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017729.264967:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334332800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017729.264975:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017729.264978:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017729.264979:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017729.264983:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1995 00000100:00080000:0.0:1734017729.264986:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017729.264989:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017729.264993:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017729.265001:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334332928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017729.265009:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017729.265013:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017729.265014:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017729.265018:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 1995 00000100:00080000:0.0:1734017729.265020:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017729.265024:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017729.265025:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334332416:192.168.201.116@tcp:38:kworker.0 00000008:00100000:0.0:1734017729.265028:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017729.265035:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334333056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017729.265046:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334332544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017729.265069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334332672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017729.265085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334332800:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017729.265099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334332928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017729.265114:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334333056:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017729.804763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017730.806759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017731.808754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017732.270730:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017732.270778:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017732.270797:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017732.270832:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017732.270840:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017732.270846:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017732.270854:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017732.270860:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017732.270866:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734017732.270951:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017729/real 1734017732] req@ffff8800a934fb80 x1818247334332672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017732.270985:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334332672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017732.271002:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017732.271007:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017732.271015:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334332672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017732.271027:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017729/real 1734017732] req@ffff8800a934d500 x1818247334332800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017732.271048:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334332800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017732.271060:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017732.271064:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017732.271070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334332800:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017732.271138:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017729/real 1734017732] req@ffff8800a934df80 x1818247334332928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017732.271160:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334332928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017732.271174:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017732.271178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017732.271194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334332928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017732.271204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017729/real 1734017732] req@ffff8800a934ce00 x1818247334333056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017732.271223:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334333056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017732.271236:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017732.271239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017732.271244:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334333056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017732.271258:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017729/real 1734017732] req@ffff8800a934e680 x1818247334332288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017732.271276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334332288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017732.271287:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017732.271292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017732.271297:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334332288:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017732.271307:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017729/real 1734017732] req@ffff8800a934e300 x1818247334332416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017732.271323:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334332416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017732.271336:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017732.271340:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017732.271346:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334332416:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017732.271356:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017729/real 1734017732] req@ffff8800a934f100 x1818247334332544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017732.271373:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334332544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017732.271385:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017732.271389:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017732.271395:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334332544:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017732.810742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017733.812725:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017734.272641:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017734.272654:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017734.272659:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017734.272673:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2000 00000100:00080000:0.0:1734017734.272685:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017734.272695:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017734.272744:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334333184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017734.272784:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017734.272788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017734.272790:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017734.272795:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2000 00000100:00080000:0.0:1734017734.272799:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017734.272803:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017734.272814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017734.272826:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334333312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017734.272838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017734.272841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017734.272843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017734.272846:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2000 00000100:00080000:0.0:1734017734.272849:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017734.272854:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017734.272858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017734.272867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334333440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017734.272876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017734.272880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017734.272881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017734.272885:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2000 00000100:00080000:0.0:1734017734.272887:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017734.272891:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017734.272895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017734.272903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334333568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017734.272913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017734.272917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734017734.272917:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334333184:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734017734.272918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017734.272923:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2000 00000100:00080000:0.0:1734017734.272926:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017734.272930:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017734.272934:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017734.272946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334333696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017734.272954:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017734.272957:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017734.272958:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017734.272962:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2000 00000100:00080000:0.0:1734017734.272975:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017734.272979:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017734.272983:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017734.272990:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734017734.272993:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334333824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017734.273001:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017734.273004:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017734.273006:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017734.273009:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2000 00000100:00080000:0.0:1734017734.273012:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017734.273014:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017734/real 1734017734] req@ffff8800a934ea00 x1818247334333184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017734.273016:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017734.273020:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017734.273028:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334333952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017734.273034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334333184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017734.273045:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017734.273077:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017734.273083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334333184:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017734.273092:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334333312:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017734.273112:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017734.273120:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017734/real 1734017734] req@ffff8800a934c380 x1818247334333312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017734.273132:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334333312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017734.273141:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017734.273144:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017734.273147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334333312:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017734.273154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334333440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017734.273168:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017734.273175:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017734/real 1734017734] req@ffff8800a934ca80 x1818247334333440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017734.273185:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334333440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017734.273193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017734.273196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017734.273199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334333440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017734.273225:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334333568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017734.273239:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017734.273245:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017734/real 1734017734] req@ffff8800a934ed80 x1818247334333568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017734.273255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334333568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017734.273263:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017734.273266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017734.273269:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334333568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017734.273275:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334333696:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017734.273287:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017734.273293:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017734/real 1734017734] req@ffff8800a934c700 x1818247334333696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017734.273303:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334333696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017734.273310:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017734.273313:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017734.273316:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334333696:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017734.273323:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334333824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017734.273335:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017734.273342:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017734/real 1734017734] req@ffff8800a934d880 x1818247334333824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017734.273352:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334333824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017734.273359:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017734.273361:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017734.273364:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334333824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017734.273370:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334333952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017734.273383:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017734.273389:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017734/real 1734017734] req@ffff8800a934dc00 x1818247334333952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017734.273398:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334333952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017734.273405:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017734.273408:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017734.273411:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334333952:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017734.814750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017735.816741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017736.818659:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017737.820733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017738.822765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017739.280604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017739.280618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017739.280623:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017739.280636:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2005 00000100:00080000:0.0:1734017739.280647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017739.280657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017739.280701:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334334080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017739.280741:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017739.280745:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017739.280747:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017739.280752:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2005 00000100:00080000:0.0:1734017739.280756:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017739.280760:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017739.280771:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017739.280783:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334334208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017739.280792:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017739.280796:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017739.280798:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017739.280802:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2005 00000100:00080000:0.0:1734017739.280805:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017739.280808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017739.280814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017739.280825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334334336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017739.280835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017739.280839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017739.280841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017739.280844:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2005 00000100:00080000:0.0:1734017739.280847:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017739.280850:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017739.280855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017739.280863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334334464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017739.280873:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017739.280876:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017739.280878:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017739.280881:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2005 00000100:00080000:0.0:1734017739.280884:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017739.280887:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017739.280888:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334334080:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734017739.280892:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017739.280900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334334592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017739.280908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017739.280911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017739.280914:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017739.280918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2005 00000100:00080000:0.0:1734017739.280920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017739.280924:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017739.280929:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017739.280937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334334720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017739.280945:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017739.280948:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017739.280950:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017739.280953:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2005 00000100:00080000:0.0:1734017739.280956:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017739.280960:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017739.280987:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734017739.280995:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017739.281004:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334334848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017739.281012:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017739/real 1734017739] req@ffff8800a934d180 x1818247334334080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017739.281033:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334334080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017739.281045:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017739.281049:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017739.281054:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334334080:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017739.281062:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334334208:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017739.281083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017739.281090:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017739/real 1734017739] req@ffff8800a934c000 x1818247334334208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017739.281102:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334334208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017739.281109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017739.281112:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017739.281116:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334334208:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017739.281122:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334334336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017739.281136:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017739.281142:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017739/real 1734017739] req@ffff8800a934dc00 x1818247334334336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017739.281152:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334334336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017739.281159:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017739.281163:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017739.281166:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334334336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017739.281194:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334334464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017739.281208:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017739.281214:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017739/real 1734017739] req@ffff8800a934d880 x1818247334334464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017739.281225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334334464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017739.281232:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017739.281234:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017739.281237:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334334464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017739.281243:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334334592:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017739.281256:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017739.281262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017739/real 1734017739] req@ffff8800a934c700 x1818247334334592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017739.281272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334334592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017739.281279:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017739.281282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017739.281285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334334592:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017739.281291:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334334720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017739.281307:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017739.281313:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017739/real 1734017739] req@ffff8800a934ed80 x1818247334334720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017739.281323:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334334720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017739.281330:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017739.281333:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017739.281336:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334334720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017739.281342:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334334848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017739.281354:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017739.281360:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017739/real 1734017739] req@ffff8800a934ca80 x1818247334334848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017739.281370:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334334848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017739.281377:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017739.281379:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017739.281382:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334334848:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017739.824782:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017740.826764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017741.828778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017742.830734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017743.832721:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017744.288702:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017744.288711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017744.288716:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017744.288726:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2010 00000100:00080000:2.0:1734017744.288735:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017744.288743:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017744.288783:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334334976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017744.288822:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017744.288826:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017744.288828:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017744.288833:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2010 00000100:00080000:2.0:1734017744.288836:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017744.288840:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017744.288850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017744.288862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334335104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017744.288872:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017744.288875:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017744.288914:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017744.288919:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2010 00000100:00080000:2.0:1734017744.288922:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017744.288926:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017744.288931:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017744.288941:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334335232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017744.288952:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017744.288955:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017744.288957:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334334976:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017744.288957:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017744.288960:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2010 00000100:00080000:2.0:1734017744.288962:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017744.288965:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017744.288969:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017744.288977:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334335360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017744.288986:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017744.288989:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017744.288990:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017744.288993:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2010 00000100:00080000:2.0:1734017744.288996:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017744.288999:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017744.289003:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017744.289010:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334335488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017744.289018:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017744.289021:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1734017744.289023:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017744.289023:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017744.289027:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2010 00000100:00080000:2.0:1734017744.289030:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017744.289033:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017744.289037:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017744.289044:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017744/real 1734017744] req@ffff8800a78a1880 x1818247334334976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017744.289045:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334335616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017744.289052:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017744.289055:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017744.289057:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017744.289060:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2010 00000100:00080000:2.0:1734017744.289063:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017744.289066:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017744.289069:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017744.289071:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334334976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017744.289077:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334335744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017744.289082:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017744.289085:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017744.289090:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334334976:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017744.289097:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334335104:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017744.289113:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017744.289120:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017744/real 1734017744] req@ffff8800a78a3b80 x1818247334335104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017744.289130:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334335104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017744.289138:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017744.289140:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017744.289144:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334335104:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017744.289167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334335232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017744.289181:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017744.289187:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017744/real 1734017744] req@ffff8800a78a3480 x1818247334335232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017744.289197:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334335232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017744.289204:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017744.289208:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017744.289211:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334335232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017744.289217:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334335360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017744.289230:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017744.289236:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017744/real 1734017744] req@ffff8800a78a1f80 x1818247334335360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017744.289245:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334335360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017744.289253:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017744.289255:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017744.289258:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334335360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017744.289263:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334335488:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017744.289276:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017744.289282:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017744/real 1734017744] req@ffff8800a78a0a80 x1818247334335488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017744.289291:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334335488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017744.289298:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017744.289300:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017744.289303:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334335488:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017744.289309:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334335616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017744.289325:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017744.289331:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017744/real 1734017744] req@ffff8800a78a2680 x1818247334335616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017744.289342:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334335616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017744.289349:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017744.289351:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017744.289354:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334335616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017744.289359:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334335744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017744.289371:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017744.289377:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017744/real 1734017744] req@ffff8800a78a0700 x1818247334335744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017744.289387:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334335744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017744.289394:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017744.289396:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017744.289399:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334335744:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017744.834726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017745.836736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017746.838744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017747.840688:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017748.842717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017749.296611:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017749.296621:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017749.296625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017749.296636:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2015 00000100:00080000:2.0:1734017749.296645:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017749.296653:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017749.296691:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334335872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017749.296734:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017749.296738:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017749.296740:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017749.296744:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2015 00000100:00080000:2.0:1734017749.296747:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017749.296751:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017749.296761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017749.296773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334336000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017749.296782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017749.296785:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017749.296787:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017749.296814:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2015 00000100:00080000:2.0:1734017749.296817:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017749.296821:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017749.296827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017749.296836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334336128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017749.296845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017749.296848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017749.296850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017749.296853:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2015 00000100:00080000:2.0:1734017749.296856:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017749.296860:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017749.296864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017749.296865:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334335872:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017749.296872:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334336256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017749.296881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017749.296884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017749.296885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017749.296888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2015 00000100:00080000:2.0:1734017749.296891:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017749.296894:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017749.296898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017749.296905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334336384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017749.296913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017749.296916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017749.296917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017749.296921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2015 00000100:00080000:2.0:1734017749.296924:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017749.296927:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017749.296929:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017749.296931:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017749.296939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334336512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017749.296946:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017749.296950:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017749/real 1734017749] req@ffff8800a78a0700 x1818247334335872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017749.296950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017749.296951:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017749.296955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2015 00000100:00080000:2.0:1734017749.296958:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017749.296961:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017749.296965:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017749.296972:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334336640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017749.296978:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334335872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017749.296988:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017749.296992:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017749.296996:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334335872:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017749.297003:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334336000:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017749.297020:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017749.297027:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017749/real 1734017749] req@ffff8800a78a2680 x1818247334336000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017749.297038:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334336000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017749.297046:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017749.297048:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017749.297051:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334336000:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017749.297057:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334336128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017749.297070:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017749.297077:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017749/real 1734017749] req@ffff8800a78a0a80 x1818247334336128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017749.297087:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334336128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017749.297094:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017749.297097:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017749.297100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334336128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017749.297128:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334336256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017749.297141:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017749.297148:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017749/real 1734017749] req@ffff8800a78a1f80 x1818247334336256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017749.297158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334336256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017749.297165:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017749.297167:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017749.297170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334336256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017749.297175:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334336384:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017749.297188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017749.297194:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017749/real 1734017749] req@ffff8800a78a3480 x1818247334336384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017749.297204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334336384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017749.297211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017749.297213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017749.297216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334336384:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017749.297222:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334336512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017749.297237:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017749.297244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017749/real 1734017749] req@ffff8800a78a3b80 x1818247334336512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017749.297254:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334336512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017749.297261:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017749.297263:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017749.297266:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334336512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017749.297271:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334336640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017749.297284:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017749.297290:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017749/real 1734017749] req@ffff8800a78a1880 x1818247334336640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017749.297300:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334336640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017749.297307:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017749.297309:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017749.297312:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334336640:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017749.844834:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017750.846740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017751.848742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017752.850727:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017753.852679:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017754.304615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017754.304625:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017754.304630:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017754.304640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2020 00000100:00080000:2.0:1734017754.304648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017754.304656:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017754.304693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334336768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017754.304768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017754.304772:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017754.304775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017754.304779:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2020 00000100:00080000:2.0:1734017754.304783:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017754.304787:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017754.304797:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017754.304810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334336896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017754.304821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017754.304824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017754.304825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017754.304829:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2020 00000100:00080000:2.0:1734017754.304832:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017754.304835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017754.304840:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017754.304849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334337024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017754.304857:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017754.304860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017754.304862:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017754.304865:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2020 00000100:00080000:2.0:1734017754.304867:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017754.304870:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017754.304874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017754.304882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334337152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017754.304887:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334336768:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017754.304891:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017754.304894:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017754.304896:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017754.304899:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2020 00000100:00080000:2.0:1734017754.304902:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017754.304905:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017754.304909:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017754.304917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334337280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017754.304925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017754.304927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017754.304929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017754.304932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2020 00000100:00080000:2.0:1734017754.304935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017754.304938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017754.304942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017754.304949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334337408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017754.304952:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017754.304957:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017754.304959:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017754.304961:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017754.304964:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2020 00000100:00080000:2.0:1734017754.304969:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017754.304972:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017754.304974:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017754/real 1734017754] req@ffff8800a78a1880 x1818247334336768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734017754.304976:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017754.304984:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334337536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017754.305001:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334336768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017754.305011:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017754.305015:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017754.305019:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334336768:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017754.305027:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334336896:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017754.305043:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017754.305050:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017754/real 1734017754] req@ffff8800a78a3b80 x1818247334336896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017754.305062:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334336896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017754.305069:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017754.305072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017754.305075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334336896:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017754.305081:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334337024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017754.305094:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017754.305100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017754/real 1734017754] req@ffff8800a78a3480 x1818247334337024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017754.305110:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334337024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017754.305117:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017754.305121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017754.305124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334337024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017754.305148:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334337152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017754.305161:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017754.305168:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017754/real 1734017754] req@ffff8800a78a1f80 x1818247334337152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017754.305178:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334337152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017754.305185:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017754.305188:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017754.305191:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334337152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017754.305196:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334337280:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017754.305208:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017754.305214:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017754/real 1734017754] req@ffff8800a78a0a80 x1818247334337280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017754.305224:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334337280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017754.305231:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017754.305233:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017754.305236:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334337280:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017754.305242:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334337408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017754.305258:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017754.305264:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017754/real 1734017754] req@ffff8800a78a2680 x1818247334337408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017754.305274:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334337408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017754.305281:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017754.305283:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017754.305286:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334337408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017754.305292:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334337536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017754.305303:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017754.305310:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017754/real 1734017754] req@ffff8800a78a0700 x1818247334337536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017754.305319:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334337536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017754.305326:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017754.305328:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017754.305331:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334337536:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017754.854772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017755.856748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017756.858733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017757.860728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017758.862785:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017759.312718:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017759.312727:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017759.312732:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017759.312743:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2025 00000100:00080000:0.0:1734017759.312751:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017759.312759:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017759.312796:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334337664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017759.312843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017759.312847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017759.312849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017759.312853:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2025 00000100:00080000:0.0:1734017759.312857:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017759.312861:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017759.312870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017759.312881:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334337792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017759.312890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017759.312893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017759.312895:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017759.312899:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2025 00000100:00080000:0.0:1734017759.312902:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017759.312905:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017759.312910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017759.312918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334337920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017759.312926:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017759.312930:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017759.312931:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017759.312935:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2025 00000100:00080000:0.0:1734017759.312937:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017759.312941:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017759.312945:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017759.312953:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334337664:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734017759.312954:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334338048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017759.312963:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017759.312966:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017759.312967:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017759.312970:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2025 00000100:00080000:0.0:1734017759.312973:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017759.312976:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017759.312980:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017759.312988:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334338176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017759.312996:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017759.312999:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017759.313001:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017759.313004:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2025 00000100:00080000:0.0:1734017759.313007:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017759.313010:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017759.313014:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017759.313019:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734017759.313023:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334338304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017759.313031:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017759.313035:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017759.313036:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017759.313040:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2025 00000100:00100000:1.0:1734017759.313041:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017759/real 1734017759] req@ffff8800a934c380 x1818247334337664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017759.313043:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017759.313046:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017759.313050:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017759.313057:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334338432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017759.313070:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334337664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017759.313079:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017759.313082:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017759.313087:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334337664:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017759.313094:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334337792:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017759.313111:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017759.313118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017759/real 1734017759] req@ffff8800a934ea00 x1818247334337792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017759.313129:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334337792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017759.313136:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017759.313139:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017759.313142:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334337792:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017759.313148:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334337920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017759.313161:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017759.313167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017759/real 1734017759] req@ffff8800a934f100 x1818247334337920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017759.313177:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334337920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017759.313184:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017759.313188:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017759.313191:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334337920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017759.313218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334338048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017759.313231:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017759.313237:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017759/real 1734017759] req@ffff8800a934e300 x1818247334338048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017759.313248:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334338048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017759.313255:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017759.313258:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017759.313260:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334338048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017759.313266:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334338176:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017759.313278:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017759.313284:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017759/real 1734017759] req@ffff8800a934e680 x1818247334338176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017759.313294:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334338176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017759.313301:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017759.313303:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017759.313306:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334338176:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017759.313313:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334338304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017759.313328:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017759.313334:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017759/real 1734017759] req@ffff8800a934ce00 x1818247334338304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017759.313344:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334338304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017759.313351:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017759.313354:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017759.313356:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334338304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017759.313362:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334338432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017759.313374:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017759.313380:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017759/real 1734017759] req@ffff8800a934df80 x1818247334338432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017759.313389:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334338432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017759.313396:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017759.313398:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017759.313401:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334338432:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017759.864601:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017760.866762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017761.868798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017762.870769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017763.872743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017764.320632:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017764.320645:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017764.320650:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017764.320665:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2030 00000100:00080000:0.0:1734017764.320676:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017764.320685:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017764.320734:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334338560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017764.320774:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017764.320779:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017764.320781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017764.320786:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2030 00000100:00080000:0.0:1734017764.320789:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017764.320793:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017764.320804:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017764.320816:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334338688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017764.320828:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017764.320831:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017764.320833:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017764.320836:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2030 00000100:00080000:0.0:1734017764.320839:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017764.320843:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017764.320848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017764.320860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334338816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017764.320869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017764.320872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017764.320874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017764.320877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2030 00000100:00080000:0.0:1734017764.320880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017764.320883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017764.320887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017764.320896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334338944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017764.320906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017764.320909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017764.320911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017764.320914:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2030 00000100:00100000:1.0:1734017764.320916:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334338560:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734017764.320918:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017764.320921:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017764.320926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017764.320935:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334339072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017764.320943:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017764.320946:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017764.320948:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017764.320951:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2030 00000100:00080000:0.0:1734017764.320954:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017764.320957:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017764.320961:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017764.320970:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334339200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017764.320978:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017764.320981:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017764.320982:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017764.320986:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2030 00000100:00080000:0.0:1734017764.320989:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017764.320991:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734017764.320993:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017764.320997:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017764.321006:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334339328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017764.321016:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017764/real 1734017764] req@ffff8800a934d500 x1818247334338560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017764.321045:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334338560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017764.321056:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017764.321060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017764.321065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334338560:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017764.321073:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334338688:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017764.321090:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017764.321097:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017764/real 1734017764] req@ffff8800a934fb80 x1818247334338688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017764.321108:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334338688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017764.321116:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017764.321118:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017764.321122:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334338688:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017764.321128:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334338816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017764.321142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017764.321148:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017764/real 1734017764] req@ffff8800a934df80 x1818247334338816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017764.321158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334338816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017764.321166:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017764.321169:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017764.321172:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334338816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017764.321198:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334338944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017764.321212:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017764.321219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017764/real 1734017764] req@ffff8800a934ce00 x1818247334338944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017764.321229:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334338944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017764.321236:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017764.321239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017764.321242:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334338944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017764.321248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334339072:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017764.321261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017764.321267:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017764/real 1734017764] req@ffff8800a934e680 x1818247334339072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017764.321276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334339072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017764.321284:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017764.321286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017764.321289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334339072:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017764.321295:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334339200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017764.321311:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017764.321317:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017764/real 1734017764] req@ffff8800a934e300 x1818247334339200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017764.321328:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334339200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017764.321335:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017764.321337:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017764.321340:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334339200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017764.321345:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334339328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017764.321358:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017764.321364:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017764/real 1734017764] req@ffff8800a934f100 x1818247334339328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017764.321374:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334339328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017764.321381:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017764.321383:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017764.321386:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334339328:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017764.874754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017765.876743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017766.878748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017767.880722:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017768.882720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017769.328701:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017769.328711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017769.328715:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017769.328725:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2035 00000100:00080000:2.0:1734017769.328734:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017769.328741:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017769.328781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334339456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017769.328819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017769.328824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017769.328826:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017769.328830:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2035 00000100:00080000:2.0:1734017769.328833:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017769.328837:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017769.328847:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017769.328859:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334339584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017769.328868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017769.328871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017769.328873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017769.328878:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2035 00000100:00080000:2.0:1734017769.328880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017769.328884:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017769.328889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017769.328898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334339712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017769.328907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017769.328910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017769.328912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017769.328915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2035 00000100:00080000:2.0:1734017769.328917:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017769.328920:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017769.328924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017769.328930:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334339456:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017769.328933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334339840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017769.328942:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017769.328945:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017769.328946:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017769.328949:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2035 00000100:00080000:2.0:1734017769.328952:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017769.328955:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017769.328959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017769.328966:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334339968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017769.328975:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017769.328978:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017769.328979:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017769.328983:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2035 00000100:00080000:2.0:1734017769.328986:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017769.328989:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017769.328993:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017769.328996:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017769.329001:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334340096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017769.329009:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017769.329012:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017769.329013:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017769.329016:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2035 00000100:00100000:1.0:1734017769.329018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017769/real 1734017769] req@ffff8800a78a0700 x1818247334339456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017769.329019:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017769.329022:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017769.329026:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017769.329033:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334340224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017769.329045:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334339456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017769.329055:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017769.329058:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017769.329063:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334339456:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017769.329071:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334339584:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017769.329087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017769.329094:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017769/real 1734017769] req@ffff8800a78a2680 x1818247334339584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017769.329105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334339584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017769.329112:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017769.329115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017769.329118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334339584:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017769.329124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334339712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017769.329138:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017769.329144:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017769/real 1734017769] req@ffff8800a78a0a80 x1818247334339712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017769.329154:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334339712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017769.329161:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017769.329164:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017769.329167:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334339712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017769.329192:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334339840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017769.329205:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017769.329212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017769/real 1734017769] req@ffff8800a78a1f80 x1818247334339840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017769.329222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334339840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017769.329229:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017769.329231:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017769.329234:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334339840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017769.329240:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334339968:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017769.329252:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017769.329258:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017769/real 1734017769] req@ffff8800a78a3480 x1818247334339968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017769.329268:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334339968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017769.329275:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017769.329277:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017769.329280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334339968:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017769.329286:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334340096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017769.329301:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017769.329307:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017769/real 1734017769] req@ffff8800a78a3b80 x1818247334340096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017769.329318:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334340096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017769.329325:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017769.329327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017769.329330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334340096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017769.329335:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334340224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017769.329347:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017769.329353:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017769/real 1734017769] req@ffff8800a78a1880 x1818247334340224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017769.329363:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334340224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017769.329370:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017769.329372:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017769.329375:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334340224:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017769.884610:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017770.886760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017771.888716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017772.890771:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017773.892756:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017774.336608:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017774.336622:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017774.336627:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017774.336641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2040 00000100:00080000:2.0:1734017774.336652:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017774.336661:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017774.336712:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334340352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017774.336759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017774.336764:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017774.336766:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017774.336770:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2040 00000100:00080000:2.0:1734017774.336774:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017774.336778:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017774.336789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017774.336802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334340480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017774.336811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017774.336814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017774.336816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017774.336820:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2040 00000100:00080000:2.0:1734017774.336823:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017774.336828:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017774.336832:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017774.336842:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334340608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017774.336851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017774.336854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017774.336856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017774.336860:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2040 00000100:00080000:2.0:1734017774.336862:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017774.336866:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017774.336871:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017774.336880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334340736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017774.336889:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017774.336892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017774.336893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017774.336895:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334340352:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017774.336897:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2040 00000100:00080000:2.0:1734017774.336900:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017774.336904:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017774.336908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017774.336917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334340864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017774.336925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017774.336928:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017774.336929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017774.336933:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2040 00000100:00080000:2.0:1734017774.336935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017774.336939:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017774.336943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017774.336951:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334340992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017774.336958:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017774.336962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017774.336964:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017774.336966:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017774.336968:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2040 00000100:00080000:2.0:1734017774.336972:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017774.336975:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017774.336979:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017774.336987:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334341120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017774.336990:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017774/real 1734017774] req@ffff8800a78a1880 x1818247334340352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017774.337017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334340352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017774.337029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017774.337032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017774.337037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334340352:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017774.337045:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334340480:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017774.337062:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017774.337069:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017774/real 1734017774] req@ffff8800a78a3b80 x1818247334340480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017774.337081:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334340480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017774.337088:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017774.337091:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017774.337095:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334340480:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017774.337101:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334340608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017774.337115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017774.337121:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017774/real 1734017774] req@ffff8800a78a3480 x1818247334340608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017774.337131:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334340608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017774.337139:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017774.337142:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017774.337145:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334340608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017774.337172:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334340736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017774.337185:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017774.337192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017774/real 1734017774] req@ffff8800a78a1f80 x1818247334340736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017774.337202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334340736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017774.337210:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017774.337212:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017774.337215:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334340736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017774.337222:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334340864:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017774.337234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017774.337240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017774/real 1734017774] req@ffff8800a78a0a80 x1818247334340864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017774.337250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334340864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017774.337258:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017774.337260:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017774.337263:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334340864:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017774.337269:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334340992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017774.337285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017774.337291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017774/real 1734017774] req@ffff8800a78a2680 x1818247334340992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017774.337302:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334340992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017774.337309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017774.337311:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017774.337314:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334340992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017774.337320:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334341120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017774.337333:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017774.337339:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017774/real 1734017774] req@ffff8800a78a0700 x1818247334341120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017774.337349:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334341120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017774.337355:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017774.337358:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017774.337361:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334341120:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017774.894712:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017775.896751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017776.898740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017777.900732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017778.902719:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017779.344606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017779.344615:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017779.344620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017779.344630:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2045 00000100:00080000:2.0:1734017779.344639:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017779.344647:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017779.344684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334341248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017779.344720:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017779.344724:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017779.344726:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017779.344731:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2045 00000100:00080000:2.0:1734017779.344734:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017779.344737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017779.344748:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017779.344759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334341376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017779.344770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017779.344773:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017779.344775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017779.344779:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2045 00000100:00080000:2.0:1734017779.344782:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017779.344785:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017779.344790:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017779.344799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334341504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017779.344807:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017779.344811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017779.344812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017779.344816:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2045 00000100:00080000:2.0:1734017779.344818:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017779.344821:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017779.344825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017779.344831:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334341248:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017779.344833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334341632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017779.344842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017779.344845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017779.344846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017779.344850:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2045 00000100:00080000:2.0:1734017779.344853:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017779.344856:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017779.344860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017779.344867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334341760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017779.344875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017779.344878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017779.344879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017779.344883:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2045 00000100:00080000:2.0:1734017779.344886:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017779.344889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017779.344893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017779.344898:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017779.344901:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334341888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017779.344908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017779.344911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017779.344913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017779.344916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2045 00000100:00080000:2.0:1734017779.344918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017779.344920:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017779/real 1734017779] req@ffff8800a78a0700 x1818247334341248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017779.344922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017779.344926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017779.344933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334342016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017779.344947:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334341248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017779.344957:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017779.344960:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017779.344965:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334341248:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017779.344973:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334341376:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017779.344989:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017779.344995:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017779/real 1734017779] req@ffff8800a78a2680 x1818247334341376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017779.345006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334341376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017779.345014:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017779.345017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017779.345020:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334341376:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017779.345026:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334341504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017779.345041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017779.345047:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017779/real 1734017779] req@ffff8800a78a0a80 x1818247334341504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017779.345057:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334341504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017779.345064:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017779.345068:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017779.345071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334341504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017779.345095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334341632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017779.345108:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017779.345114:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017779/real 1734017779] req@ffff8800a78a1f80 x1818247334341632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017779.345125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334341632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017779.345132:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017779.345134:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017779.345137:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334341632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017779.345143:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334341760:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017779.345155:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017779.345161:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017779/real 1734017779] req@ffff8800a78a3480 x1818247334341760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017779.345171:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334341760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017779.345178:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017779.345180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017779.345183:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334341760:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017779.345188:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334341888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017779.345203:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017779.345209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017779/real 1734017779] req@ffff8800a78a3b80 x1818247334341888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017779.345219:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334341888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017779.345226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017779.345228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017779.345231:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334341888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017779.345237:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334342016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017779.345249:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017779.345255:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017779/real 1734017779] req@ffff8800a78a1880 x1818247334342016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017779.345264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334342016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017779.345271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017779.345273:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017779.345276:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334342016:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017779.904741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017780.906698:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017781.908795:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017782.910759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017783.912738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017784.352609:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017784.352618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017784.352622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017784.352632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2050 00000100:00080000:2.0:1734017784.352642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017784.352650:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017784.352689:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334342144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017784.352724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017784.352728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017784.352730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017784.352734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2050 00000100:00080000:2.0:1734017784.352738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017784.352742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017784.352752:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017784.352763:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334342272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017784.352773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017784.352775:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017784.352777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017784.352781:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2050 00000100:00080000:2.0:1734017784.352784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017784.352788:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017784.352793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017784.352801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334342400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017784.352810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017784.352813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017784.352815:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017784.352818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2050 00000100:00080000:2.0:1734017784.352820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017784.352823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017784.352827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017784.352833:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334342144:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017784.352836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334342528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017784.352845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017784.352847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017784.352849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017784.352852:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2050 00000100:00080000:2.0:1734017784.352854:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017784.352858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017784.352862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017784.352869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334342656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017784.352878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017784.352881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017784.352882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017784.352886:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2050 00000100:00080000:2.0:1734017784.352889:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017784.352892:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017784.352896:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017784.352899:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017784.352903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334342784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017784.352911:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017784.352914:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017784.352915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017784.352919:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2050 00000100:00100000:1.0:1734017784.352921:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017784/real 1734017784] req@ffff8800a78a1880 x1818247334342144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017784.352921:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017784.352925:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017784.352928:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017784.352936:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334342912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017784.352948:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334342144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017784.352958:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017784.352961:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017784.352966:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334342144:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017784.352973:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334342272:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017784.352990:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017784.352997:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017784/real 1734017784] req@ffff8800a78a3b80 x1818247334342272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017784.353008:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334342272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017784.353016:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017784.353018:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017784.353022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334342272:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017784.353027:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334342400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017784.353041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017784.353048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017784/real 1734017784] req@ffff8800a78a3480 x1818247334342400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017784.353058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334342400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017784.353065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017784.353068:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017784.353071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334342400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017784.353099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334342528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017784.353113:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017784.353119:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017784/real 1734017784] req@ffff8800a78a1f80 x1818247334342528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017784.353130:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334342528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017784.353137:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017784.353139:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017784.353142:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334342528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017784.353148:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334342656:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017784.353160:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017784.353166:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017784/real 1734017784] req@ffff8800a78a0a80 x1818247334342656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017784.353176:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334342656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017784.353183:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017784.353185:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017784.353188:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334342656:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017784.353239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334342784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017784.353261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017784.353269:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017784/real 1734017784] req@ffff8800a78a2680 x1818247334342784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017784.353280:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334342784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017784.353288:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017784.353290:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017784.353293:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334342784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017784.353299:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334342912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017784.353311:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017784.353318:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017784/real 1734017784] req@ffff8800a78a0700 x1818247334342912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017784.353328:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334342912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017784.353335:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017784.353337:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017784.353340:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334342912:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017784.914734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017785.916755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017786.918750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017787.920769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017788.922727:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017789.360538:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017789.360548:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017789.360553:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017789.360565:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2055 00000100:00080000:2.0:1734017789.360575:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017789.360583:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017789.360621:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334343040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017789.360660:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017789.360664:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017789.360666:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017789.360670:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2055 00000100:00080000:2.0:1734017789.360674:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017789.360678:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017789.360688:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017789.360699:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334343168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017789.360709:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017789.360712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017789.360713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017789.360717:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2055 00000100:00080000:2.0:1734017789.360720:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017789.360724:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017789.360729:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017789.360737:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334343296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017789.360746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017789.360749:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017789.360750:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017789.360754:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2055 00000100:00080000:2.0:1734017789.360756:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017789.360760:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017789.360764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017789.360772:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334343040:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017789.360772:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334343424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017789.360781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017789.360784:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017789.360786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017789.360789:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2055 00000100:00080000:2.0:1734017789.360791:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017789.360794:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017789.360798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017789.360805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334343552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017789.360813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017789.360816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017789.360818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017789.360821:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2055 00000100:00080000:2.0:1734017789.360831:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017789.360835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017789.360838:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017789.360839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017789.360847:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334343680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017789.360856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017789.360859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017789.360860:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017789/real 1734017789] req@ffff8800a78a0700 x1818247334343040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017789.360861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017789.360865:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2055 00000100:00080000:2.0:1734017789.360868:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017789.360871:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017789.360875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017789.360882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334343808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017789.360888:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334343040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017789.360898:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017789.360901:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017789.360906:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334343040:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017789.360913:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334343168:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017789.360930:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017789.360936:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017789/real 1734017789] req@ffff8800a78a2680 x1818247334343168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017789.360947:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334343168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017789.360955:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017789.360957:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017789.360961:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334343168:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017789.360967:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334343296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017789.360982:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017789.360988:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017789/real 1734017789] req@ffff8800a78a0a80 x1818247334343296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017789.360999:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334343296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017789.361006:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017789.361008:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017789.361012:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334343296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017789.361036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334343424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017789.361049:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017789.361056:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017789/real 1734017789] req@ffff8800a78a1f80 x1818247334343424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017789.361066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334343424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017789.361073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017789.361076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017789.361079:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334343424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017789.361084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334343552:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017789.361096:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017789.361102:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017789/real 1734017789] req@ffff8800a78a3480 x1818247334343552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017789.361153:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334343552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017789.361161:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017789.361163:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017789.361166:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334343552:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017789.361172:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334343680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017789.361191:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017789.361198:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017789/real 1734017789] req@ffff8800a78a3b80 x1818247334343680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017789.361209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334343680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017789.361216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017789.361218:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017789.361221:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334343680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017789.361227:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334343808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017789.361240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017789.361246:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017789/real 1734017789] req@ffff8800a78a1880 x1818247334343808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017789.361257:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334343808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017789.361264:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017789.361266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017789.361269:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334343808:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017789.924783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017790.926730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017791.928739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017792.930764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017793.932750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017794.368611:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017794.368620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017794.368625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017794.368636:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2061 00000100:00080000:2.0:1734017794.368645:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017794.368653:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017794.368693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334343936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017794.368729:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017794.368733:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017794.368735:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017794.368740:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2061 00000100:00080000:2.0:1734017794.368743:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017794.368747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017794.368757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017794.368768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334344064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017794.368779:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017794.368782:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017794.368784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017794.368788:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2061 00000100:00080000:2.0:1734017794.368790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017794.368794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017794.368799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017794.368808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334344192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017794.368816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017794.368819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017794.368821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017794.368824:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2061 00000100:00080000:2.0:1734017794.368827:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017794.368830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017794.368834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017794.368840:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334343936:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017794.368842:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334344320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017794.368852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017794.368855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017794.368856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017794.368859:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2061 00000100:00080000:2.0:1734017794.368862:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017794.368866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017794.368870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017794.368877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334344448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017794.368885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017794.368888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017794.368890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017794.368893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2061 00000100:00080000:2.0:1734017794.368896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017794.368900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017794.368903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017794.368911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334344576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017794.368918:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017794.368921:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017794.368923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017794.368926:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2061 00000100:00080000:2.0:1734017794.368928:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017794.368932:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334344064:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017794.368932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017794.368936:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017794.368943:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334344704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017794.368953:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334344192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017794.368975:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334344320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017794.368990:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334344448:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017794.369005:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334344576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017794.369020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334344704:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017794.934753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017795.936729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017796.938729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017797.374751:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017797.374794:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017797.374813:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017797.374847:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017797.374855:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017797.374861:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017797.374868:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017797.374874:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017797.374880:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734017797.374949:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017794/real 1734017797] req@ffff8800a78a1f80 x1818247334344320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017797.375020:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334344320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017797.375038:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017797.375045:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017797.375052:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334344320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017797.375064:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017794/real 1734017797] req@ffff8800a78a0a80 x1818247334344448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017797.375083:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334344448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017797.375096:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017797.375100:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017797.375106:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334344448:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017797.375115:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017794/real 1734017797] req@ffff8800a78a2680 x1818247334344576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017797.375134:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334344576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017797.375147:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017797.375151:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017797.375156:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334344576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017797.375176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017794/real 1734017797] req@ffff8800a78a0700 x1818247334344704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017797.375195:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334344704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017797.375207:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017797.375211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017797.375216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334344704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017797.375226:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017794/real 1734017797] req@ffff8800a78a1880 x1818247334343936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017797.375244:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334343936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017797.375257:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017797.375260:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017797.375266:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334343936:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017797.375276:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017794/real 1734017797] req@ffff8800a78a3b80 x1818247334344064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017797.375295:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334344064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017797.375318:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017797.375323:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017797.375328:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334344064:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017797.375338:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017794/real 1734017797] req@ffff8800a78a3480 x1818247334344192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017797.375356:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334344192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017797.375369:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017797.375373:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017797.375379:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334344192:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017797.940722:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017798.942728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017799.376582:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017799.376592:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017799.376597:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017799.376607:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2066 00000100:00080000:2.0:1734017799.376616:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017799.376623:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017799.376661:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334344832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017799.376696:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017799.376700:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017799.376702:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017799.376706:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2066 00000100:00080000:2.0:1734017799.376709:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017799.376713:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017799.376724:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017799.376736:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334344960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017799.376746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017799.376749:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017799.376751:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017799.376755:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2066 00000100:00080000:2.0:1734017799.376758:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017799.376761:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017799.376766:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017799.376776:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334345088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017799.376785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017799.376788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017799.376790:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017799.376793:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2066 00000100:00080000:2.0:1734017799.376795:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017799.376798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017799.376802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017799.376810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334345216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017799.376811:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334344832:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017799.376819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017799.376822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017799.376824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017799.376827:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2066 00000100:00080000:2.0:1734017799.376829:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017799.376833:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017799.376837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017799.376844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334345344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017799.376853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017799.376856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017799.376858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017799.376861:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2066 00000100:00080000:2.0:1734017799.376873:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017799.376877:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017799.376879:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017799.376881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017799.376889:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334345472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017799.376897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017799.376900:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017799/real 1734017799] req@ffff8800a78a3480 x1818247334344832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017799.376900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017799.376902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017799.376905:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2066 00000100:00080000:2.0:1734017799.376908:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017799.376911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017799.376915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017799.376920:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334344832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017799.376923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334345600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017799.376931:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017799.376934:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017799.376966:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334344832:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017799.376975:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334344960:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017799.376995:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017799.377004:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017799/real 1734017799] req@ffff8800a78a3b80 x1818247334344960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017799.377016:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334344960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017799.377024:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017799.377026:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017799.377030:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334344960:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017799.377036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334345088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017799.377049:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017799.377055:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017799/real 1734017799] req@ffff8800a78a1880 x1818247334345088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017799.377066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334345088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017799.377073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017799.377075:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017799.377078:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334345088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017799.377102:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334345216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017799.377115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017799.377122:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017799/real 1734017799] req@ffff8800a78a0700 x1818247334345216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017799.377132:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334345216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017799.377147:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017799.377150:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017799.377153:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334345216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017799.377158:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334345344:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017799.377171:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017799.377177:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017799/real 1734017799] req@ffff8800a78a2680 x1818247334345344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017799.377187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334345344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017799.377194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017799.377196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017799.377199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334345344:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017799.377204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334345472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017799.377216:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017799.377221:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017799/real 1734017799] req@ffff8800a78a0a80 x1818247334345472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017799.377231:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334345472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017799.377238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017799.377241:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017799.377244:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334345472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017799.377250:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334345600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017799.377262:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017799.377268:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017799/real 1734017799] req@ffff8800a78a1f80 x1818247334345600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017799.377278:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334345600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017799.377285:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017799.377287:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017799.377290:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334345600:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017799.944748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017800.946755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017801.948738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017802.950748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017803.952653:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017804.384597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017804.384606:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017804.384611:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017804.384622:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2071 00000100:00080000:2.0:1734017804.384631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017804.384639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017804.384677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334345728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017804.384719:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017804.384723:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017804.384725:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017804.384729:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2071 00000100:00080000:2.0:1734017804.384733:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017804.384737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017804.384747:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017804.384759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334345856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017804.384769:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017804.384771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017804.384773:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017804.384777:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2071 00000100:00080000:2.0:1734017804.384780:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017804.384783:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017804.384788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017804.384797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334345984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017804.384805:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017804.384808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017804.384809:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017804.384812:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2071 00000100:00080000:2.0:1734017804.384815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017804.384819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017804.384823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017804.384831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334346112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017804.384835:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334345728:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017804.384840:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017804.384843:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017804.384845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017804.384848:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2071 00000100:00080000:2.0:1734017804.384850:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017804.384882:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017804.384887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017804.384896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334346240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017804.384904:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017804.384907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017804.384909:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017804.384912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2071 00000100:00080000:2.0:1734017804.384915:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017804.384918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017804.384923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017804.384930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334346368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017804.384938:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017804.384938:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017804.384942:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017804.384943:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017804.384947:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2071 00000100:00080000:2.0:1734017804.384950:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017804.384953:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017804.384957:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017804.384959:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017804/real 1734017804] req@ffff8800a78a1f80 x1818247334345728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017804.384965:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334346496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017804.384980:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334345728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017804.384991:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017804.384994:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017804.384999:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334345728:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017804.385006:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334345856:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017804.385023:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017804.385030:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017804/real 1734017804] req@ffff8800a78a0a80 x1818247334345856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017804.385050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334345856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017804.385058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017804.385060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017804.385064:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334345856:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017804.385069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334345984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017804.385083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017804.385089:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017804/real 1734017804] req@ffff8800a78a2680 x1818247334345984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017804.385099:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334345984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017804.385106:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017804.385109:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017804.385112:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334345984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017804.385136:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334346112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017804.385149:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017804.385156:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017804/real 1734017804] req@ffff8800a78a0700 x1818247334346112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017804.385166:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334346112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017804.385173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017804.385176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017804.385179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334346112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017804.385186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334346240:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017804.385198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017804.385204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017804/real 1734017804] req@ffff8800a78a1880 x1818247334346240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017804.385214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334346240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017804.385221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017804.385223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017804.385226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334346240:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017804.385231:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334346368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017804.385243:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017804.385249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017804/real 1734017804] req@ffff8800a78a3b80 x1818247334346368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017804.385258:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334346368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017804.385266:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017804.385269:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017804.385272:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334346368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017804.385277:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334346496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017804.385292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017804.385298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017804/real 1734017804] req@ffff8800a78a3480 x1818247334346496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017804.385308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334346496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017804.385315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017804.385317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017804.385320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334346496:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017804.954717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017805.956714:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017806.958756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017807.960711:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017808.962739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017809.392602:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017809.392612:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017809.392617:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017809.392628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2076 00000100:00080000:2.0:1734017809.392636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017809.392644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017809.392683:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334346624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017809.392717:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017809.392721:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017809.392723:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017809.392727:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2076 00000100:00080000:2.0:1734017809.392730:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017809.392734:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017809.392744:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017809.392756:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334346752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017809.392767:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017809.392807:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017809.392808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017809.392813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2076 00000100:00080000:2.0:1734017809.392816:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017809.392820:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017809.392826:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017809.392836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334346880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017809.392845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017809.392848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017809.392850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017809.392853:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2076 00000100:00080000:2.0:1734017809.392855:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017809.392859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017809.392863:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334346624:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017809.392863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017809.392870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334347008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017809.392880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017809.392883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017809.392884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017809.392887:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2076 00000100:00080000:2.0:1734017809.392890:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017809.392894:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017809.392897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017809.392904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334347136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017809.392912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017809.392915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017809.392917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017809.392920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2076 00000100:00080000:2.0:1734017809.392923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017809.392926:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017809.392928:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017809.392930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017809.392937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334347264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017809.392945:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017809.392948:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017809.392949:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017809.392950:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017809/real 1734017809] req@ffff8800a78a3480 x1818247334346624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017809.392953:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2076 00000100:00080000:2.0:1734017809.392955:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017809.392958:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017809.392962:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017809.392969:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334346624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017809.392970:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334347392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017809.392980:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017809.392983:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017809.392988:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334346624:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017809.392995:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334346752:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017809.393012:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017809.393019:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017809/real 1734017809] req@ffff8800a78a3b80 x1818247334346752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017809.393030:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334346752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017809.393039:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017809.393041:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017809.393045:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334346752:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017809.393050:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334346880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017809.393065:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017809.393071:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017809/real 1734017809] req@ffff8800a78a1880 x1818247334346880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017809.393081:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334346880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017809.393088:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017809.393091:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017809.393094:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334346880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017809.393119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334347008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017809.393132:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017809.393138:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017809/real 1734017809] req@ffff8800a78a0700 x1818247334347008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017809.393149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334347008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017809.393156:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017809.393159:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017809.393162:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334347008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017809.393168:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334347136:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017809.393181:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017809.393187:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017809/real 1734017809] req@ffff8800a78a2680 x1818247334347136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017809.393197:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334347136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017809.393204:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017809.393206:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017809.393209:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334347136:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017809.393214:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334347264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017809.393226:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017809.393232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017809/real 1734017809] req@ffff8800a78a0a80 x1818247334347264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017809.393241:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334347264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017809.393248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017809.393251:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017809.393254:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334347264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017809.393259:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334347392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017809.393271:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017809.393277:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017809/real 1734017809] req@ffff8800a78a1f80 x1818247334347392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017809.393288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334347392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017809.393296:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017809.393298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017809.393301:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334347392:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017809.964737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017810.966776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017811.968719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017812.970784:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017813.972755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017814.400611:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017814.400620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017814.400625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017814.400635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2081 00000100:00080000:2.0:1734017814.400644:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017814.400652:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017814.400722:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334347520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017814.400754:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017814.400759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017814.400761:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017814.400765:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2081 00000100:00080000:2.0:1734017814.400769:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017814.400773:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017814.400783:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017814.400795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334347648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017814.400805:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017814.400808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017814.400810:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017814.400814:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2081 00000100:00080000:2.0:1734017814.400816:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017814.400820:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017814.400825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017814.400834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334347776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017814.400843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017814.400846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017814.400847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017814.400851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2081 00000100:00080000:2.0:1734017814.400853:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017814.400856:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017814.400860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017814.400868:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334347520:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017814.400868:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334347904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017814.400877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017814.400880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017814.400881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017814.400885:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2081 00000100:00080000:2.0:1734017814.400887:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017814.400890:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017814.400894:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017814.400901:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334348032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017814.400910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017814.400913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017814.400915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017814.400918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2081 00000100:00080000:2.0:1734017814.400930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017814.400934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017814.400935:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017814.400938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017814.400946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334348160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017814.400955:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017814.400956:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017814/real 1734017814] req@ffff8800a78a1f80 x1818247334347520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017814.400958:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017814.400959:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017814.400962:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2081 00000100:00080000:2.0:1734017814.400965:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017814.400968:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017814.400972:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017814.400976:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334347520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017814.400979:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334348288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017814.400987:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017814.400990:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017814.400995:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334347520:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017814.401002:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334347648:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017814.401019:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017814.401025:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017814/real 1734017814] req@ffff8800a78a0a80 x1818247334347648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017814.401037:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334347648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017814.401045:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017814.401048:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017814.401051:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334347648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017814.401058:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334347776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017814.401071:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017814.401077:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017814/real 1734017814] req@ffff8800a78a2680 x1818247334347776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017814.401087:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334347776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017814.401095:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017814.401097:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017814.401101:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334347776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017814.401125:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334347904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017814.401138:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017814.401144:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017814/real 1734017814] req@ffff8800a78a0700 x1818247334347904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017814.401155:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334347904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017814.401163:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017814.401165:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017814.401168:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334347904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017814.401173:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334348032:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017814.401189:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017814.401195:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017814/real 1734017814] req@ffff8800a78a1880 x1818247334348032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017814.401205:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334348032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017814.401212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017814.401214:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017814.401217:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334348032:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017814.401222:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334348160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017814.401234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017814.401240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017814/real 1734017814] req@ffff8800a78a3b80 x1818247334348160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017814.401249:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334348160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017814.401256:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017814.401258:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017814.401262:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334348160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017814.401267:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334348288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017814.401279:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017814.401285:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017814/real 1734017814] req@ffff8800a78a3480 x1818247334348288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017814.401301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334348288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017814.401308:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017814.401310:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017814.401313:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334348288:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017814.974756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017815.976744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017816.978726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017817.980767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017818.982756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017819.408579:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017819.408589:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017819.408593:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017819.408624:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2086 00000100:00080000:2.0:1734017819.408633:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017819.408641:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017819.408684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334348416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017819.408709:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017819.408712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017819.408713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017819.408716:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2086 00000100:00080000:2.0:1734017819.408718:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017819.408720:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017819.408727:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017819.408734:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334348544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017819.408739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017819.408741:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017819.408742:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017819.408761:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2086 00000100:00080000:2.0:1734017819.408764:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017819.408768:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017819.408773:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017819.408782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334348672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017819.408791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017819.408794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017819.408796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017819.408799:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2086 00000100:00080000:2.0:1734017819.408801:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017819.408805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017819.408809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017819.408817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334348800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017819.408826:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334348416:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017819.408826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017819.408829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017819.408830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017819.408834:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2086 00000100:00080000:2.0:1734017819.408836:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017819.408839:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017819.408843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017819.408850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334348928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017819.408858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017819.408861:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017819.408862:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017819.408865:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2086 00000100:00080000:2.0:1734017819.408868:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017819.408871:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017819.408875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017819.408882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334349056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017819.408889:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734017819.408892:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017819.408893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017819.408895:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017819.408898:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2086 00000100:00080000:2.0:1734017819.408901:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017819.408904:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017819.408908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017819.408912:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017819/real 1734017819] req@ffff8800a78a3480 x1818247334348416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017819.408915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334349184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017819.408931:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334348416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017819.408942:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017819.408945:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017819.408950:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334348416:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017819.408967:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334348544:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017819.408983:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017819.408990:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017819/real 1734017819] req@ffff8800a78a3b80 x1818247334348544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017819.409001:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334348544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017819.409009:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017819.409013:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017819.409016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334348544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017819.409033:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334348672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017819.409046:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017819.409052:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017819/real 1734017819] req@ffff8800a78a1880 x1818247334348672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017819.409062:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334348672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017819.409070:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017819.409072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017819.409075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334348672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017819.409098:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334348800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017819.409111:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017819.409117:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017819/real 1734017819] req@ffff8800a78a0700 x1818247334348800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017819.409128:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334348800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017819.409135:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017819.409137:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017819.409140:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334348800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017819.409145:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334348928:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017819.409161:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017819.409167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017819/real 1734017819] req@ffff8800a78a2680 x1818247334348928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017819.409177:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334348928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017819.409184:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017819.409186:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017819.409189:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334348928:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017819.409194:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334349056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017819.409205:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017819.409211:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017819/real 1734017819] req@ffff8800a78a0a80 x1818247334349056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017819.409221:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334349056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017819.409228:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017819.409230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017819.409233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334349056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017819.409238:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334349184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017819.409251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017819.409257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017819/real 1734017819] req@ffff8800a78a1f80 x1818247334349184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017819.409273:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334349184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017819.409281:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017819.409283:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017819.409286:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334349184:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017819.984633:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017820.986756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017821.988736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017822.990741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017823.992752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017824.416606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017824.416615:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017824.416620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017824.416630:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2091 00000100:00080000:2.0:1734017824.416638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017824.416646:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017824.416683:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334349312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017824.416715:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017824.416719:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017824.416721:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017824.416726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2091 00000100:00080000:2.0:1734017824.416729:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017824.416733:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017824.416743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017824.416755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334349440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017824.416766:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017824.416769:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017824.416770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017824.416774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2091 00000100:00080000:2.0:1734017824.416777:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017824.416780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017824.416785:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017824.416794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334349568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017824.416803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017824.416806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017824.416808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017824.416811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2091 00000100:00080000:2.0:1734017824.416814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017824.416817:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334349312:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017824.416817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017824.416821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017824.416829:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334349696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017824.416837:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017824.416840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017824.416842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017824.416845:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2091 00000100:00080000:2.0:1734017824.416849:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017824.416852:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017824.416856:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017824.416863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334349824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017824.416871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017824.416874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017824.416875:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017824.416878:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2091 00000100:00080000:2.0:1734017824.416881:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017824.416884:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017824.416884:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017824.416888:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017824.416895:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334349952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017824.416903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017824.416906:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017824/real 1734017824] req@ffff8800a78a1f80 x1818247334349312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017824.416906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017824.416908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017824.416911:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2091 00000100:00080000:2.0:1734017824.416914:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017824.416917:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017824.416921:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017824.416925:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334349312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017824.416928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334350080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017824.416936:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017824.416939:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017824.416944:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334349312:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017824.416952:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334349440:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017824.416968:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017824.416975:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017824/real 1734017824] req@ffff8800a78a0a80 x1818247334349440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017824.416986:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334349440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017824.416993:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017824.416997:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017824.417001:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334349440:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017824.417024:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334349568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017824.417038:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017824.417044:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017824/real 1734017824] req@ffff8800a78a2680 x1818247334349568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017824.417055:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334349568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017824.417062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017824.417064:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017824.417067:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334349568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017824.417073:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334349696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017824.417085:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017824.417091:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017824/real 1734017824] req@ffff8800a78a0700 x1818247334349696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017824.417101:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334349696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017824.417108:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017824.417110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017824.417113:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334349696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017824.417119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334349824:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017824.417135:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017824.417141:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017824/real 1734017824] req@ffff8800a78a1880 x1818247334349824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017824.417151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334349824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017824.417158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017824.417160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017824.417163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334349824:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017824.417168:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334349952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017824.417180:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017824.417186:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017824/real 1734017824] req@ffff8800a78a3b80 x1818247334349952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017824.417196:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334349952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017824.417203:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017824.417205:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017824.417208:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334349952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017824.417213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334350080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017824.417226:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017824.417231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017824/real 1734017824] req@ffff8800a78a3480 x1818247334350080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017824.417248:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334350080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017824.417256:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017824.417258:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017824.417261:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334350080:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017824.994736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017825.996691:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017826.998694:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017828.000771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017829.002707:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017829.424594:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017829.424604:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017829.424609:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017829.424619:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2096 00000100:00080000:2.0:1734017829.424628:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017829.424635:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017829.424673:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334350208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017829.424706:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017829.424710:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017829.424712:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017829.424717:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2096 00000100:00080000:2.0:1734017829.424720:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017829.424724:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017829.424734:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017829.424745:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334350336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017829.424755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017829.424758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017829.424760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017829.424764:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2096 00000100:00080000:2.0:1734017829.424767:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017829.424771:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017829.424776:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017829.424785:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334350464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017829.424794:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017829.424797:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017829.424799:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017829.424802:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2096 00000100:00080000:2.0:1734017829.424804:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017829.424808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017829.424812:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334350208:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017829.424812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017829.424820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334350592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017829.424829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017829.424832:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017829.424833:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017829.424836:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2096 00000100:00080000:2.0:1734017829.424839:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017829.424842:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017829.424846:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017829.424853:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334350720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017829.424861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017829.424865:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017829.424867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017829.424870:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2096 00000100:00080000:2.0:1734017829.424873:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017829.424877:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017829.424879:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017829.424881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017829.424888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334350848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017829.424896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017829.424899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017829.424900:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017829.424901:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017829/real 1734017829] req@ffff8800a78a3480 x1818247334350208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017829.424904:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2096 00000100:00080000:2.0:1734017829.424906:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017829.424909:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017829.424913:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017829.424921:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334350208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017829.424921:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334350976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017829.424932:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017829.424935:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017829.424940:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334350208:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017829.424948:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334350336:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017829.424964:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017829.424971:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017829/real 1734017829] req@ffff8800a78a3b80 x1818247334350336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017829.424982:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334350336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017829.424990:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017829.424993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017829.424997:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334350336:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017829.425003:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334350464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017829.425017:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017829.425023:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017829/real 1734017829] req@ffff8800a78a1880 x1818247334350464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017829.425033:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334350464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017829.425041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017829.425043:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017829.425047:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334350464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017829.425070:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334350592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017829.425083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017829.425089:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017829/real 1734017829] req@ffff8800a78a0700 x1818247334350592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017829.425099:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334350592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017829.425107:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017829.425109:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017829.425112:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334350592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017829.425118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334350720:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017829.425134:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017829.425140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017829/real 1734017829] req@ffff8800a78a2680 x1818247334350720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017829.425151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334350720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017829.425158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017829.425160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017829.425163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334350720:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017829.425168:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334350848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017829.425179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017829.425185:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017829/real 1734017829] req@ffff8800a78a0a80 x1818247334350848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017829.425195:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334350848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017829.425202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017829.425204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017829.425207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334350848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017829.425213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334350976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017829.425224:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017829.425230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017829/real 1734017829] req@ffff8800a78a1f80 x1818247334350976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017829.425247:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334350976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017829.425254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017829.425256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017829.425259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334350976:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017830.004691:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017831.006750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017832.008728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017833.010726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017834.012720:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017834.432613:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017834.432623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017834.432628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017834.432639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2101 00000100:00080000:2.0:1734017834.432648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017834.432656:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017834.432694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334351104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017834.432737:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017834.432741:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017834.432743:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017834.432748:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2101 00000100:00080000:2.0:1734017834.432751:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017834.432754:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017834.432764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017834.432775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334351232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017834.432785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017834.432788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017834.432789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017834.432793:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2101 00000100:00080000:2.0:1734017834.432796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017834.432800:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017834.432805:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017834.432813:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334351360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017834.432821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017834.432824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017834.432826:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017834.432829:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2101 00000100:00080000:2.0:1734017834.432832:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017834.432835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017834.432839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017834.432848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334351488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017834.432850:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334351104:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017834.432857:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017834.432860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017834.432861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017834.432865:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2101 00000100:00080000:2.0:1734017834.432867:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017834.432870:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017834.432874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017834.432882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334351616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017834.432889:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017834.432892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017834.432894:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017834.432897:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2101 00000100:00080000:2.0:1734017834.432900:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017834.432903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017834.432907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017834.432914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334351744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017834.432919:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017834.432923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017834.432926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017834.432928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017834.432931:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2101 00000100:00080000:2.0:1734017834.432936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017834.432939:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017834.432941:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017834/real 1734017834] req@ffff8800a78a1f80 x1818247334351104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734017834.432943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017834.432951:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334351872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017834.432961:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334351104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017834.432972:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017834.432975:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017834.432980:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334351104:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017834.432987:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334351232:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017834.433004:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017834.433011:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017834/real 1734017834] req@ffff8800a78a0a80 x1818247334351232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017834.433022:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334351232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017834.433030:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017834.433034:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017834.433037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334351232:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017834.433043:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334351360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017834.433058:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017834.433064:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017834/real 1734017834] req@ffff8800a78a2680 x1818247334351360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017834.433074:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334351360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017834.433082:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017834.433084:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017834.433088:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334351360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017834.433114:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334351488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017834.433127:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017834.433133:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017834/real 1734017834] req@ffff8800a78a0700 x1818247334351488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017834.433144:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334351488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017834.433152:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017834.433154:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017834.433157:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334351488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017834.433162:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334351616:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017834.433178:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017834.433184:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017834/real 1734017834] req@ffff8800a78a1880 x1818247334351616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017834.433195:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334351616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017834.433202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017834.433204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017834.433207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334351616:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017834.433212:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334351744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017834.433224:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017834.433230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017834/real 1734017834] req@ffff8800a78a3b80 x1818247334351744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017834.433239:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334351744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017834.433247:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017834.433249:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017834.433252:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334351744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017834.433257:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334351872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017834.433269:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017834.433275:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017834/real 1734017834] req@ffff8800a78a3480 x1818247334351872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017834.433291:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334351872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017834.433298:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017834.433300:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017834.433303:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334351872:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017835.014761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017836.016732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017837.018695:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017838.020724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017839.022714:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017839.440595:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017839.440605:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017839.440609:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017839.440619:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2106 00000100:00080000:2.0:1734017839.440627:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017839.440635:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017839.440673:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334352000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017839.440711:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017839.440717:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017839.440720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017839.440727:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2106 00000100:00080000:2.0:1734017839.440731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017839.440737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017839.440751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017839.440769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334352128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017839.440781:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334352000:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017839.440786:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017839.440790:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017839.440792:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017839.440798:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2106 00000100:00080000:2.0:1734017839.440802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017839.440807:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017839.440814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017839.440827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334352256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017839.440839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017839.440843:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017839.440846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017839.440850:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017839.440850:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2106 00000100:00080000:2.0:1734017839.440854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017839.440859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017839.440865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017839.440874:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017839/real 1734017839] req@ffff8800a78a3480 x1818247334352000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017839.440876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334352384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017839.440890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017839.440895:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017839.440897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017839.440900:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334352000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017839.440903:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2106 00000100:00080000:2.0:1734017839.440908:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017839.440913:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734017839.440914:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017839.440918:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000008:00100000:2.0:1734017839.440919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734017839.440925:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334352000:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017839.440930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334352512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017839.440942:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017839.440946:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017839.440948:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017839.440952:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334352128:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017839.440953:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2106 00000100:00080000:2.0:1734017839.440958:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017839.440963:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017839.440969:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017839.440977:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017839.440982:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334352640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017839.440988:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017839/real 1734017839] req@ffff8800a78a3b80 x1818247334352128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017839.440996:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017839.441001:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017839.441004:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017839.441005:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334352128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017839.441009:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2106 00000100:00080000:2.0:1734017839.441013:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734017839.441018:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017839.441018:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734017839.441023:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000008:00100000:2.0:1734017839.441024:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017839.441028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334352128:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734017839.441036:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334352768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017839.441037:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334352256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017839.441060:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017839.441070:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017839/real 1734017839] req@ffff8800a78a1880 x1818247334352256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017839.441088:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334352256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017839.441100:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017839.441104:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017839.441109:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334352256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017839.441118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334352384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017839.441144:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017839.441156:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017839/real 1734017839] req@ffff8800a78a0700 x1818247334352384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017839.441169:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334352384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017839.441177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017839.441179:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017839.441182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334352384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017839.441189:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334352512:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017839.441208:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017839.441215:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017839/real 1734017839] req@ffff8800a78a2680 x1818247334352512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017839.441225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334352512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017839.441233:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017839.441235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017839.441238:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334352512:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017839.441255:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334352640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017839.441316:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017839.441324:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017839/real 1734017839] req@ffff8800a78a0a80 x1818247334352640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017839.441334:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334352640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017839.441342:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017839.441344:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017839.441347:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334352640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017839.441353:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334352768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017839.441365:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017839.441371:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017839/real 1734017839] req@ffff8800a78a1f80 x1818247334352768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017839.441388:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334352768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017839.441396:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017839.441398:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017839.441401:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334352768:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017840.024751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017841.026741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017842.028731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017843.030724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017844.032736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017844.448590:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017844.448600:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017844.448605:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017844.448614:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2111 00000100:00080000:2.0:1734017844.448624:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017844.448631:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017844.448669:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334352896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017844.448702:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017844.448706:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017844.448708:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017844.448713:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2111 00000100:00080000:2.0:1734017844.448716:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017844.448720:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017844.448731:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017844.448743:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334353024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017844.448752:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017844.448755:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017844.448757:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017844.448761:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2111 00000100:00080000:2.0:1734017844.448764:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017844.448767:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017844.448773:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017844.448782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334353152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017844.448791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017844.448794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017844.448796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017844.448799:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2111 00000100:00080000:2.0:1734017844.448801:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017844.448805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017844.448809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017844.448810:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334352896:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017844.448817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334353280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017844.448825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017844.448828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017844.448829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017844.448833:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2111 00000100:00080000:2.0:1734017844.448835:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017844.448838:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017844.448842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017844.448849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334353408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017844.448858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017844.448861:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017844.448862:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017844.448866:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2111 00000100:00080000:2.0:1734017844.448869:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017844.448872:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017844.448876:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017844.448876:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017844.448884:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334353536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017844.448892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017844.448895:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017844.448896:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017844.448897:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017844/real 1734017844] req@ffff8800a78a1f80 x1818247334352896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017844.448900:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2111 00000100:00080000:2.0:1734017844.448902:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017844.448906:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017844.448909:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017844.448917:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334352896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017844.448917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334353664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017844.448928:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017844.448931:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017844.448936:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334352896:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017844.448943:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334353024:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017844.448959:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017844.448966:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017844/real 1734017844] req@ffff8800a78a0a80 x1818247334353024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017844.448977:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334353024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017844.448985:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017844.448989:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017844.448992:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334353024:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017844.448999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334353152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017844.449012:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017844.449018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017844/real 1734017844] req@ffff8800a78a2680 x1818247334353152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017844.449029:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334353152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017844.449036:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017844.449039:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017844.449042:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334353152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017844.449066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334353280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017844.449079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017844.449085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017844/real 1734017844] req@ffff8800a78a0700 x1818247334353280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017844.449095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334353280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017844.449103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017844.449105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017844.449108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334353280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017844.449114:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334353408:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017844.449134:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017844.449140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017844/real 1734017844] req@ffff8800a78a1880 x1818247334353408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017844.449151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334353408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017844.449158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017844.449160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017844.449163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334353408:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017844.449169:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334353536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017844.449248:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017844.449256:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017844/real 1734017844] req@ffff8800a78a3b80 x1818247334353536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017844.449267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334353536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017844.449274:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017844.449277:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017844.449280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334353536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017844.449285:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334353664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017844.449298:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017844.449304:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017844/real 1734017844] req@ffff8800a78a3480 x1818247334353664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017844.449322:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334353664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017844.449329:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017844.449332:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017844.449335:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334353664:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017845.034731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017846.036716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017847.038738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017848.040748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017849.042710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017849.456607:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017849.456617:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017849.456622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017849.456632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2116 00000100:00080000:2.0:1734017849.456641:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017849.456648:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017849.456685:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334353792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017849.456725:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017849.456729:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017849.456731:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017849.456735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2116 00000100:00080000:2.0:1734017849.456738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017849.456742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017849.456753:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017849.456764:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334353920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017849.456773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017849.456776:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017849.456778:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017849.456782:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2116 00000100:00080000:2.0:1734017849.456784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017849.456788:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017849.456793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017849.456802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334354048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017849.456810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017849.456813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017849.456815:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017849.456818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2116 00000100:00080000:2.0:1734017849.456821:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017849.456824:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017849.456828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017849.456833:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334353792:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017849.456836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334354176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017849.456845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017849.456848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017849.456850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017849.456853:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2116 00000100:00080000:2.0:1734017849.456855:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017849.456858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017849.456862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017849.456869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334354304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017849.456877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017849.456880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017849.456882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017849.456885:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2116 00000100:00080000:2.0:1734017849.456896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017849.456900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017849.456901:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017849.456904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017849.456912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334354432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017849.456920:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017849.456922:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017849/real 1734017849] req@ffff8800a78a3480 x1818247334353792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017849.456924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017849.456926:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017849.456929:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2116 00000100:00080000:2.0:1734017849.456933:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017849.456936:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017849.456939:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017849.456942:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334353792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017849.456947:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334354560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017849.456953:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017849.456956:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017849.456961:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334353792:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017849.456968:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334353920:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017849.456985:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017849.456991:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017849/real 1734017849] req@ffff8800a78a3b80 x1818247334353920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017849.457002:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334353920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017849.457010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017849.457013:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017849.457017:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334353920:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017849.457023:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334354048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017849.457037:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017849.457043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017849/real 1734017849] req@ffff8800a78a1880 x1818247334354048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017849.457053:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334354048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017849.457061:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017849.457063:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017849.457066:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334354048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017849.457142:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334354176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017849.457160:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017849.457167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017849/real 1734017849] req@ffff8800a78a0700 x1818247334354176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017849.457179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334354176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017849.457187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017849.457189:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017849.457192:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334354176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017849.457198:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334354304:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017849.457214:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017849.457220:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017849/real 1734017849] req@ffff8800a78a2680 x1818247334354304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017849.457231:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334354304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017849.457238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017849.457240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017849.457243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334354304:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017849.457248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334354432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017849.457260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017849.457266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017849/real 1734017849] req@ffff8800a78a0a80 x1818247334354432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017849.457276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334354432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017849.457284:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017849.457286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017849.457289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334354432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017849.457294:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334354560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017849.457306:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017849.457312:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017849/real 1734017849] req@ffff8800a78a1f80 x1818247334354560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017849.457329:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334354560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017849.457336:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017849.457338:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017849.457341:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334354560:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017850.044764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017851.046704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017852.048695:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017853.050689:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017854.052741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017854.464585:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017854.464595:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017854.464600:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017854.464610:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2121 00000100:00080000:2.0:1734017854.464618:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017854.464626:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017854.464665:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334354688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017854.464698:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017854.464702:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017854.464705:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017854.464709:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2121 00000100:00080000:2.0:1734017854.464712:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017854.464716:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017854.464726:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017854.464738:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334354816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017854.464749:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017854.464752:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017854.464754:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017854.464758:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2121 00000100:00080000:2.0:1734017854.464760:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017854.464764:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017854.464768:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017854.464777:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334354944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017854.464785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017854.464788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017854.464790:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017854.464793:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2121 00000100:00080000:2.0:1734017854.464796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017854.464799:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017854.464803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017854.464804:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334354688:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017854.464811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334355072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017854.464820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017854.464823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017854.464825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017854.464828:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2121 00000100:00080000:2.0:1734017854.464831:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017854.464834:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017854.464838:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017854.464846:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334355200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017854.464853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017854.464857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017854.464858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017854.464861:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2121 00000100:00080000:2.0:1734017854.464864:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017854.464867:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017854.464868:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017854.464872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017854.464879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334355328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017854.464887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017854.464890:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017854/real 1734017854] req@ffff8800a78a1f80 x1818247334354688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017854.464890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017854.464891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017854.464895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2121 00000100:00080000:2.0:1734017854.464898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017854.464901:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017854.464905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017854.464909:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334354688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017854.464912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334355456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017854.464920:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017854.464924:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017854.464929:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334354688:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017854.464936:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334354816:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017854.464953:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017854.464960:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017854/real 1734017854] req@ffff8800a78a0a80 x1818247334354816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017854.464971:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334354816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017854.464979:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017854.464982:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017854.464986:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334354816:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017854.464992:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334354944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017854.465052:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017854.465060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017854/real 1734017854] req@ffff8800a78a2680 x1818247334354944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017854.465071:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334354944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017854.465078:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017854.465081:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017854.465084:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334354944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017854.465109:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334355072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017854.465123:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017854.465130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017854/real 1734017854] req@ffff8800a78a0700 x1818247334355072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017854.465141:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334355072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017854.465148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017854.465150:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017854.465154:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334355072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017854.465159:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334355200:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017854.465175:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017854.465181:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017854/real 1734017854] req@ffff8800a78a1880 x1818247334355200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017854.465192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334355200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017854.465200:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017854.465202:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017854.465205:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334355200:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017854.465210:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334355328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017854.465222:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017854.465228:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017854/real 1734017854] req@ffff8800a78a3b80 x1818247334355328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017854.465237:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334355328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017854.465245:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017854.465247:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017854.465250:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334355328:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017854.465255:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334355456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017854.465267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017854.465273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017854/real 1734017854] req@ffff8800a78a3480 x1818247334355456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017854.465294:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334355456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017854.465301:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017854.465303:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017854.465306:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334355456:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017855.054764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017856.056741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017857.058744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017858.060790:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017859.062786:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017859.472591:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017859.472604:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017859.472609:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017859.472622:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2126 00000100:00080000:2.0:1734017859.472633:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017859.472643:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017859.472690:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334355584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017859.472729:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017859.472734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017859.472736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017859.472741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2126 00000100:00080000:2.0:1734017859.472744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017859.472748:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017859.472759:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017859.472771:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334355712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017859.472781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017859.472785:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017859.472787:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017859.472791:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2126 00000100:00080000:2.0:1734017859.472794:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017859.472797:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017859.472803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017859.472813:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334355840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017859.472823:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017859.472826:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017859.472828:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017859.472831:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2126 00000100:00080000:2.0:1734017859.472834:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017859.472837:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017859.472841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017859.472850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334355968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017859.472859:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017859.472862:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017859.472864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017859.472868:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2126 00000100:00080000:2.0:1734017859.472870:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017859.472871:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334355584:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017859.472874:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017859.472878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017859.472887:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334356096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017859.472897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017859.472900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017859.472902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017859.472906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2126 00000100:00080000:2.0:1734017859.472908:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017859.472912:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017859.472959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017859.472969:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334356224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017859.472978:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017859.472982:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017859.472984:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017859.472987:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2126 00000100:00080000:2.0:1734017859.472991:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017859.472994:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017859.472998:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017859.473002:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334355712:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734017859.473008:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334356352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017859.473025:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334355840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017859.473051:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334355968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017859.473066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334356096:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017859.473083:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334356224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017859.473098:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334356352:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017860.064719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017861.066720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017862.068763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017862.478746:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017862.478793:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017862.478813:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017862.478847:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017862.478855:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017862.478889:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017862.478902:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017862.478912:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017862.478923:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734017862.478974:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017859/real 1734017862] req@ffff8800a78a0700 x1818247334355968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017862.479014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334355968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017862.479032:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017862.479038:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017862.479046:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334355968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017862.479058:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017859/real 1734017862] req@ffff8800a78a2680 x1818247334356096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017862.479079:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334356096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017862.479092:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017862.479096:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017862.479102:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334356096:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017862.479112:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017859/real 1734017862] req@ffff8800a78a0a80 x1818247334356224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017862.479130:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334356224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017862.479142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017862.479158:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017862.479164:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334356224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017862.479175:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017859/real 1734017862] req@ffff8800a78a1f80 x1818247334356352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017862.479194:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334356352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017862.479206:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017862.479211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017862.479217:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334356352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017862.479231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017859/real 1734017862] req@ffff8800a78a3480 x1818247334355584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017862.479250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334355584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017862.479262:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017862.479266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017862.479271:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334355584:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017862.479282:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017859/real 1734017862] req@ffff8800a78a3b80 x1818247334355712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017862.479300:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334355712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017862.479313:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017862.479317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017862.479336:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334355712:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017862.479347:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017859/real 1734017862] req@ffff8800a78a1880 x1818247334355840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017862.479367:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334355840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017862.479379:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017862.479382:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017862.479387:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334355840:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017863.070743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017864.072687:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017864.480527:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017864.480537:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017864.480542:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017864.480552:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2131 00000100:00080000:2.0:1734017864.480561:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017864.480568:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017864.480606:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334356480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017864.480646:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017864.480650:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017864.480652:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017864.480656:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2131 00000100:00080000:2.0:1734017864.480660:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017864.480664:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017864.480673:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017864.480684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334356608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017864.480694:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017864.480697:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017864.480699:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017864.480703:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2131 00000100:00080000:2.0:1734017864.480706:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017864.480710:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017864.480715:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017864.480723:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334356736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017864.480731:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017864.480734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017864.480735:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017864.480739:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2131 00000100:00080000:2.0:1734017864.480741:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017864.480745:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017864.480749:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017864.480750:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334356480:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017864.480756:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334356864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017864.480765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017864.480767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017864.480769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017864.480772:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2131 00000100:00080000:2.0:1734017864.480774:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017864.480778:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017864.480781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017864.480788:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334356992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017864.480796:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017864.480798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017864.480800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017864.480803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2131 00000100:00080000:2.0:1734017864.480806:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017864.480809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017864.480813:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017864.480813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017864.480820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334357120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017864.480828:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017864.480857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017864.480859:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017864/real 1734017864] req@ffff8800a78a1880 x1818247334356480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017864.480859:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017864.480863:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2131 00000100:00080000:2.0:1734017864.480866:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017864.480869:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017864.480874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017864.480896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334357248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017864.480902:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334356480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017864.480909:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017864.480911:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017864.480914:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334356480:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017864.480919:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334356608:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017864.480931:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017864.480936:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017864/real 1734017864] req@ffff8800a78a3b80 x1818247334356608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017864.480943:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334356608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017864.480948:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017864.480949:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017864.480951:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334356608:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017864.480955:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334356736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017864.480964:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017864.480967:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017864/real 1734017864] req@ffff8800a78a3480 x1818247334356736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017864.480973:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334356736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017864.480983:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017864.480985:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017864.480987:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334356736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017864.481003:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334356864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017864.481011:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017864.481015:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017864/real 1734017864] req@ffff8800a78a1f80 x1818247334356864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017864.481021:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334356864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017864.481026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017864.481027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017864.481029:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334356864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017864.481033:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334356992:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017864.481040:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017864.481043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017864/real 1734017864] req@ffff8800a78a0a80 x1818247334356992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017864.481049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334356992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017864.481054:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017864.481055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017864.481057:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334356992:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017864.481061:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334357120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017864.481068:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017864.481072:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017864/real 1734017864] req@ffff8800a78a2680 x1818247334357120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017864.481078:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334357120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017864.481082:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017864.481084:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017864.481085:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334357120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017864.481105:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334357248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017864.481119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017864.481125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017864/real 1734017864] req@ffff8800a78a0700 x1818247334357248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017864.481135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334357248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017864.481142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017864.481144:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017864.481147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334357248:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017865.074759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017866.076748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017867.078751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017868.080746:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017869.082745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017869.488586:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017869.488596:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017869.488600:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017869.488612:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2136 00000100:00080000:2.0:1734017869.488620:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017869.488628:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017869.488665:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334357376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017869.488698:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017869.488702:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017869.488704:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017869.488708:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2136 00000100:00080000:2.0:1734017869.488711:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017869.488715:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017869.488724:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017869.488735:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334357504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017869.488791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017869.488795:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017869.488796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017869.488801:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2136 00000100:00080000:2.0:1734017869.488804:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017869.488808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017869.488813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017869.488823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334357632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017869.488824:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334357376:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017869.488833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017869.488836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017869.488838:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017869.488841:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2136 00000100:00080000:2.0:1734017869.488844:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017869.488847:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017869.488851:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017869.488859:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334357760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017869.488867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017869.488870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017869.488872:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017869.488875:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2136 00000100:00080000:2.0:1734017869.488879:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017869.488883:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017869.488886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734017869.488890:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017869.488894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334357888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017869.488902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017869.488905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017869.488907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017869.488910:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017869/real 1734017869] req@ffff8800a78a0700 x1818247334357376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017869.488910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2136 00000100:00080000:2.0:1734017869.488913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017869.488916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017869.488920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017869.488927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334358016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017869.488930:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334357376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017869.488935:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017869.488938:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017869.488939:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734017869.488941:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017869.488943:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2136 00000100:00080000:1.0:1734017869.488944:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734017869.488945:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017869.488948:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017869.488950:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334357376:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017869.488952:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017869.488957:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334357504:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734017869.488961:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334358144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017869.488974:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017869.488981:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017869/real 1734017869] req@ffff8800a78a2680 x1818247334357504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017869.488993:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334357504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017869.489000:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017869.489003:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017869.489006:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334357504:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017869.489028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334357632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017869.489042:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017869.489048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017869/real 1734017869] req@ffff8800a78a0a80 x1818247334357632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017869.489059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334357632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017869.489066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017869.489069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017869.489073:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334357632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017869.489079:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334357760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017869.489091:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017869.489097:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017869/real 1734017869] req@ffff8800a78a1f80 x1818247334357760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017869.489107:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334357760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017869.489114:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017869.489116:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017869.489120:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334357760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017869.489126:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334357888:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017869.489138:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017869.489144:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017869/real 1734017869] req@ffff8800a78a3480 x1818247334357888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017869.489154:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334357888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017869.489161:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017869.489163:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017869.489166:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334357888:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017869.489171:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334358016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017869.489186:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017869.489192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017869/real 1734017869] req@ffff8800a78a3b80 x1818247334358016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017869.489202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334358016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017869.489209:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017869.489211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017869.489214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334358016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017869.489220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334358144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017869.489231:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017869.489237:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017869/real 1734017869] req@ffff8800a78a1880 x1818247334358144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017869.489247:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334358144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017869.489254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017869.489256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017869.489259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334358144:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017870.084794:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017871.086813:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017872.088777:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017873.090767:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017874.092750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017874.496607:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017874.496616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017874.496621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017874.496632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2141 00000100:00080000:2.0:1734017874.496641:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017874.496649:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017874.496723:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334358272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017874.496757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017874.496761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017874.496764:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017874.496768:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2141 00000100:00080000:2.0:1734017874.496771:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017874.496776:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017874.496786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017874.496798:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334358400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017874.496808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017874.496811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017874.496813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017874.496817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2141 00000100:00080000:2.0:1734017874.496819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017874.496823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017874.496828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017874.496837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334358528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017874.496846:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017874.496849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017874.496851:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017874.496854:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2141 00000100:00080000:2.0:1734017874.496856:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017874.496859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017874.496863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017874.496871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334358656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017874.496878:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334358272:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017874.496880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017874.496883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017874.496884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017874.496888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2141 00000100:00080000:2.0:1734017874.496890:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017874.496894:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017874.496897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017874.496904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334358784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017874.496913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017874.496916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017874.496918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017874.496921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2141 00000100:00080000:2.0:1734017874.496924:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017874.496927:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017874.496931:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017874.496939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334358912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017874.496942:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017874.496946:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017874.496949:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017874.496951:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017874.496954:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2141 00000100:00080000:2.0:1734017874.496959:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017874.496962:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017874.496963:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017874/real 1734017874] req@ffff8800a78a1880 x1818247334358272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734017874.496966:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017874.496973:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334359040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017874.496990:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334358272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017874.497001:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017874.497004:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017874.497009:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334358272:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017874.497016:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334358400:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017874.497032:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017874.497039:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017874/real 1734017874] req@ffff8800a78a3b80 x1818247334358400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017874.497050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334358400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017874.497058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017874.497060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017874.497064:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334358400:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017874.497070:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334358528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017874.497083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017874.497089:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017874/real 1734017874] req@ffff8800a78a3480 x1818247334358528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017874.497099:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334358528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017874.497107:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017874.497110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017874.497113:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334358528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017874.497140:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334358656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017874.497153:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017874.497159:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017874/real 1734017874] req@ffff8800a78a1f80 x1818247334358656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017874.497169:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334358656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017874.497177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017874.497179:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017874.497182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334358656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017874.497188:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334358784:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017874.497200:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017874.497206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017874/real 1734017874] req@ffff8800a78a0a80 x1818247334358784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017874.497216:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334358784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017874.497223:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017874.497225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017874.497228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334358784:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017874.497234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334358912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017874.497248:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017874.497254:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017874/real 1734017874] req@ffff8800a78a2680 x1818247334358912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017874.497265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334358912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017874.497272:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017874.497274:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017874.497277:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334358912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017874.497282:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334359040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017874.497294:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017874.497300:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017874/real 1734017874] req@ffff8800a78a0700 x1818247334359040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017874.497310:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334359040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017874.497317:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017874.497319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017874.497321:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334359040:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017875.094724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017876.096797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017877.098778:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017878.100798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017879.102765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017879.504606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017879.504616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017879.504620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017879.504630:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2146 00000100:00080000:2.0:1734017879.504638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017879.504646:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017879.504681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334359168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017879.504724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017879.504728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017879.504729:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017879.504734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2146 00000100:00080000:2.0:1734017879.504737:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017879.504741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017879.504750:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017879.504762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334359296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017879.504771:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017879.504774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017879.504776:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017879.504779:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2146 00000100:00080000:2.0:1734017879.504782:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017879.504786:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017879.504790:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017879.504799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334359424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017879.504807:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017879.504810:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017879.504812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017879.504815:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2146 00000100:00080000:2.0:1734017879.504818:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017879.504821:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017879.504825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017879.504834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334359552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017879.504835:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334359168:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017879.504843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017879.504846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017879.504847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017879.504850:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2146 00000100:00080000:2.0:1734017879.504852:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017879.504856:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017879.504860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017879.504867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334359680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017879.504875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017879.504877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017879.504879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017879.504882:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2146 00000100:00080000:2.0:1734017879.504885:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017879.504889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017879.504892:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017879.504900:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017879.504900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334359808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017879.504908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017879.504911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017879.504913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017879.504916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2146 00000100:00080000:2.0:1734017879.504919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017879.504921:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017879/real 1734017879] req@ffff8800a78a0700 x1818247334359168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017879.504923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017879.504926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017879.504933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334359936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017879.504948:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334359168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017879.504958:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017879.504961:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017879.504966:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334359168:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017879.504974:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334359296:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017879.504990:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017879.504997:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017879/real 1734017879] req@ffff8800a78a2680 x1818247334359296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017879.505008:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334359296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017879.505016:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017879.505018:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017879.505021:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334359296:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017879.505027:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334359424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017879.505040:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017879.505046:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017879/real 1734017879] req@ffff8800a78a0a80 x1818247334359424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017879.505056:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334359424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017879.505063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017879.505066:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017879.505069:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334359424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017879.505092:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334359552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017879.505105:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017879.505111:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017879/real 1734017879] req@ffff8800a78a1f80 x1818247334359552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017879.505121:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334359552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017879.505128:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017879.505130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017879.505133:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334359552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017879.505139:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334359680:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017879.505151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017879.505157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017879/real 1734017879] req@ffff8800a78a3480 x1818247334359680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017879.505167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334359680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017879.505174:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017879.505176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017879.505179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334359680:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017879.505184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334359808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017879.505199:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017879.505205:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017879/real 1734017879] req@ffff8800a78a3b80 x1818247334359808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017879.505215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334359808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017879.505222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017879.505224:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017879.505227:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334359808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017879.505232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334359936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017879.505244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017879.505249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017879/real 1734017879] req@ffff8800a78a1880 x1818247334359936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017879.505259:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334359936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017879.505266:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017879.505268:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017879.505271:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334359936:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017880.104607:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017881.106794:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017882.108795:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017883.110780:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017884.112752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017884.512615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017884.512627:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017884.512633:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017884.512646:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2151 00000100:00080000:2.0:1734017884.512657:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017884.512666:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017884.512715:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334360064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017884.512753:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017884.512758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017884.512760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017884.512764:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2151 00000100:00080000:2.0:1734017884.512768:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017884.512771:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017884.512782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017884.512796:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334360192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017884.512806:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017884.512810:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017884.512812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017884.512815:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2151 00000100:00080000:2.0:1734017884.512818:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017884.512822:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017884.512826:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017884.512836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334360320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017884.512845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017884.512849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017884.512850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017884.512854:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2151 00000100:00080000:2.0:1734017884.512857:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017884.512860:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017884.512864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017884.512873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334360448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017884.512883:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017884.512886:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734017884.512887:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334360064:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017884.512887:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017884.512891:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2151 00000100:00080000:2.0:1734017884.512895:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017884.512899:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017884.512903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017884.512912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334360576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017884.512920:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017884.512923:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017884.512924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017884.512928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2151 00000100:00080000:2.0:1734017884.512930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017884.512934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017884.512938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017884.512946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334360704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017884.512954:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017884.512957:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017884.512958:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017884.512961:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017884.512962:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2151 00000100:00080000:2.0:1734017884.512965:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017884.512969:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017884.512973:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017884.512981:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334360832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017884.512984:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017884/real 1734017884] req@ffff8800a78a1880 x1818247334360064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017884.513012:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334360064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017884.513022:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017884.513026:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017884.513031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334360064:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017884.513040:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334360192:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017884.513058:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017884.513065:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017884/real 1734017884] req@ffff8800a78a3b80 x1818247334360192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017884.513076:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334360192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017884.513084:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017884.513087:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017884.513090:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334360192:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017884.513097:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334360320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017884.513110:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017884.513117:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017884/real 1734017884] req@ffff8800a78a3480 x1818247334360320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017884.513127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334360320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017884.513135:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017884.513138:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017884.513141:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334360320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017884.513168:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334360448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017884.513182:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017884.513188:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017884/real 1734017884] req@ffff8800a78a1f80 x1818247334360448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017884.513199:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334360448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017884.513206:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017884.513208:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017884.513211:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334360448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017884.513218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334360576:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017884.513230:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017884.513236:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017884/real 1734017884] req@ffff8800a78a0a80 x1818247334360576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017884.513247:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334360576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017884.513254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017884.513256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017884.513259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334360576:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017884.513266:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334360704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017884.513281:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017884.513287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017884/real 1734017884] req@ffff8800a78a2680 x1818247334360704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017884.513297:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334360704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017884.513305:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017884.513307:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017884.513310:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334360704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017884.513316:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334360832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017884.513329:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017884.513334:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017884/real 1734017884] req@ffff8800a78a0700 x1818247334360832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017884.513344:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334360832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017884.513351:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017884.513353:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017884.513356:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334360832:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017885.114742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017886.116716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017887.118773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017888.120767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017889.122798:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017889.520590:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017889.520602:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017889.520608:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017889.520621:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2156 00000100:00080000:2.0:1734017889.520631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017889.520641:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017889.520690:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334360960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017889.520728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017889.520732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017889.520735:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017889.520740:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2156 00000100:00080000:2.0:1734017889.520743:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017889.520747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017889.520758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017889.520771:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334361088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017889.520781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017889.520784:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017889.520786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017889.520791:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2156 00000100:00080000:2.0:1734017889.520793:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017889.520797:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017889.520802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017889.520811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334361216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017889.520821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017889.520825:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017889.520826:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017889.520830:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2156 00000100:00080000:2.0:1734017889.520832:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017889.520836:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017889.520840:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017889.520848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334361344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017889.520858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017889.520861:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017889.520862:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017889.520863:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334360960:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017889.520866:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2156 00000100:00080000:2.0:1734017889.520868:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017889.520872:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017889.520876:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017889.520885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334361472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017889.520893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017889.520897:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017889.520899:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017889.520903:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2156 00000100:00080000:2.0:1734017889.520905:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017889.520909:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017889.520913:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017889.520921:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334361600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017889.520929:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017889.520932:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017889.520934:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017889.520937:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017889.520938:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2156 00000100:00080000:2.0:1734017889.520941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017889.520945:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017889.520949:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017889.520957:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334361728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017889.520961:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017889/real 1734017889] req@ffff8800a78a0700 x1818247334360960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017889.520989:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334360960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017889.521000:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017889.521004:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017889.521009:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334360960:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017889.521017:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334361088:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017889.521035:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017889.521042:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017889/real 1734017889] req@ffff8800a78a2680 x1818247334361088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017889.521053:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334361088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017889.521060:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017889.521063:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017889.521067:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334361088:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017889.521073:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334361216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017889.521087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017889.521093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017889/real 1734017889] req@ffff8800a78a0a80 x1818247334361216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017889.521103:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334361216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017889.521111:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017889.521114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017889.521117:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334361216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017889.521144:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334361344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017889.521157:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017889.521164:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017889/real 1734017889] req@ffff8800a78a1f80 x1818247334361344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017889.521174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334361344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017889.521182:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017889.521184:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017889.521187:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334361344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017889.521193:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334361472:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017889.521206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017889.521212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017889/real 1734017889] req@ffff8800a78a3480 x1818247334361472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017889.521222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334361472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017889.521229:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017889.521232:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017889.521235:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334361472:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017889.521241:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334361600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017889.521255:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017889.521261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017889/real 1734017889] req@ffff8800a78a3b80 x1818247334361600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017889.521271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334361600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017889.521278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017889.521280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017889.521283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334361600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017889.521289:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334361728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017889.521302:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017889.521308:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017889/real 1734017889] req@ffff8800a78a1880 x1818247334361728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017889.521318:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334361728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017889.521324:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017889.521327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017889.521330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334361728:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017890.124758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017891.126795:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017892.128760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017893.130623:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017894.132720:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017894.528587:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017894.528596:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017894.528601:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017894.528611:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2161 00000100:00080000:2.0:1734017894.528620:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017894.528628:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017894.528667:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334361856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017894.528711:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017894.528717:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017894.528720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017894.528726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2161 00000100:00080000:2.0:1734017894.528731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017894.528737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017894.528751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017894.528769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334361984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017894.528781:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334361856:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017894.528785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017894.528789:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017894.528791:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017894.528797:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2161 00000100:00080000:2.0:1734017894.528802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017894.528807:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017894.528814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017894.528827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334362112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017894.528839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017894.528843:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017894.528845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017894.528850:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2161 00000800:00000100:1.0:1734017894.528852:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017894.528854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017894.528859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017894.528865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017894.528876:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017894/real 1734017894] req@ffff8800a78a1880 x1818247334361856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017894.528877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334362240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017894.528891:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017894.528896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017894.528899:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017894.528903:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2161 00000100:00080000:2.0:1734017894.528907:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017894.528910:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334361856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017894.528912:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017894.528918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734017894.528924:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017894.528929:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734017894.528930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334362368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017894.528935:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334361856:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017894.528943:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017894.528947:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017894.528950:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017894.528955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2161 00000100:00080000:2.0:1734017894.528959:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017894.528962:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334361984:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017894.528964:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017894.528971:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017894.528983:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334362496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017894.528987:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017894.528998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017894/real 1734017894] req@ffff8800a78a3b80 x1818247334361984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017894.528998:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017894.529003:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017894.529006:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017894.529011:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2161 00000100:00100000:1.0:1734017894.529016:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334361984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017894.529016:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017894.529021:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017894.529027:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017894.529028:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017894.529032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017894.529037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334361984:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734017894.529038:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334362624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017894.529047:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334362112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017894.529070:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017894.529081:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017894/real 1734017894] req@ffff8800a78a3480 x1818247334362112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017894.529098:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334362112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017894.529110:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017894.529115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017894.529120:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334362112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017894.529130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334362240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017894.529156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017894.529165:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017894/real 1734017894] req@ffff8800a78a1f80 x1818247334362240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017894.529176:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334362240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017894.529184:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017894.529187:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017894.529190:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334362240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017894.529195:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334362368:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017894.529210:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017894.529216:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017894/real 1734017894] req@ffff8800a78a0a80 x1818247334362368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017894.529226:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334362368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017894.529233:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017894.529236:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017894.529239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334362368:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017894.529256:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334362496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017894.529271:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017894.529278:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017894/real 1734017894] req@ffff8800a78a2680 x1818247334362496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017894.529288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334362496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017894.529296:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017894.529298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017894.529301:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334362496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017894.529306:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334362624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017894.529359:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017894.529368:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017894/real 1734017894] req@ffff8800a78a0700 x1818247334362624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017894.529379:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334362624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017894.529386:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017894.529389:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017894.529392:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334362624:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017895.134750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017896.136763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017897.138780:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017898.140765:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017899.142783:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017899.536579:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017899.536593:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017899.536597:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017899.536608:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2166 00000100:00080000:2.0:1734017899.536617:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017899.536625:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017899.536663:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334362752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017899.536699:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017899.536703:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017899.536705:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017899.536710:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2166 00000100:00080000:2.0:1734017899.536713:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017899.536717:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017899.536726:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017899.536737:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334362880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017899.536748:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017899.536751:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017899.536753:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017899.536757:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2166 00000100:00080000:2.0:1734017899.536759:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017899.536763:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017899.536768:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017899.536776:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334363008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017899.536784:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017899.536787:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017899.536789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017899.536792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2166 00000100:00080000:2.0:1734017899.536794:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017899.536797:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017899.536802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017899.536809:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334362752:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017899.536810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334363136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017899.536819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017899.536822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017899.536823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017899.536826:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2166 00000100:00080000:2.0:1734017899.536829:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017899.536833:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017899.536836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017899.536843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334363264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017899.536851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017899.536854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017899.536855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017899.536859:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2166 00000100:00080000:2.0:1734017899.536871:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017899.536875:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017899.536875:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017899.536879:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017899.536886:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334363392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017899.536895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017899.536896:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017899/real 1734017899] req@ffff8800a78a0700 x1818247334362752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017899.536898:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017899.536899:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017899.536902:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2166 00000100:00080000:2.0:1734017899.536906:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017899.536909:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017899.536913:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017899.536920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334363520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017899.536923:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334362752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017899.536933:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017899.536936:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017899.536940:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334362752:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017899.536948:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334362880:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017899.536963:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017899.536970:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017899/real 1734017899] req@ffff8800a78a2680 x1818247334362880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017899.536981:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334362880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017899.536989:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017899.536991:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017899.536995:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334362880:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017899.537000:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334363008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017899.537018:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017899.537024:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017899/real 1734017899] req@ffff8800a78a0a80 x1818247334363008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017899.537034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334363008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017899.537041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017899.537044:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017899.537047:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334363008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017899.537070:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334363136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017899.537083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017899.537090:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017899/real 1734017899] req@ffff8800a78a1f80 x1818247334363136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017899.537100:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334363136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017899.537108:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017899.537110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017899.537113:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334363136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017899.537118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334363264:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017899.537130:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017899.537136:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017899/real 1734017899] req@ffff8800a78a3480 x1818247334363264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017899.537146:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334363264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017899.537153:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017899.537155:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017899.537158:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334363264:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017899.537164:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334363392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017899.537179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017899.537185:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017899/real 1734017899] req@ffff8800a78a3b80 x1818247334363392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017899.537196:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334363392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017899.537203:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017899.537205:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017899.537208:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334363392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017899.537213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334363520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017899.537225:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017899.537231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017899/real 1734017899] req@ffff8800a78a1880 x1818247334363520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017899.537289:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334363520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017899.537297:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017899.537299:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017899.537302:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334363520:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017900.144731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017901.146725:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017902.148765:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017903.150760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017904.152723:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017904.544601:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017904.544610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017904.544615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017904.544625:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2171 00000100:00080000:2.0:1734017904.544634:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017904.544642:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017904.544680:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334363648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017904.544714:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017904.544718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017904.544721:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017904.544725:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2171 00000100:00080000:2.0:1734017904.544728:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017904.544732:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017904.544742:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017904.544754:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334363776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017904.544763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017904.544767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017904.544768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017904.544773:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2171 00000100:00080000:2.0:1734017904.544775:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017904.544779:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017904.544785:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017904.544793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334363904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017904.544803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017904.544806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017904.544808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017904.544811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2171 00000100:00080000:2.0:1734017904.544814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017904.544815:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334363648:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017904.544817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017904.544821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017904.544829:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334364032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017904.544838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017904.544841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017904.544843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017904.544846:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2171 00000100:00080000:2.0:1734017904.544848:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017904.544851:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017904.544855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017904.544862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334364160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017904.544870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017904.544873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017904.544875:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017904.544879:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2171 00000800:00000100:1.0:1734017904.544880:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017904.544882:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017904.544885:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017904.544889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017904.544896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334364288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017904.544901:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017904/real 1734017904] req@ffff8800a78a1880 x1818247334363648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017904.544904:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017904.544907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017904.544909:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017904.544912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2171 00000100:00080000:2.0:1734017904.544915:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017904.544918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017904.544921:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017904.544928:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334363648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017904.544929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334364416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017904.544939:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017904.544942:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017904.544947:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334363648:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017904.544954:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334363776:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017904.544970:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017904.544978:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017904/real 1734017904] req@ffff8800a78a3b80 x1818247334363776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017904.544989:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334363776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017904.544996:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017904.544999:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017904.545002:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334363776:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017904.545024:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334363904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017904.545038:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017904.545044:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017904/real 1734017904] req@ffff8800a78a3480 x1818247334363904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017904.545055:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334363904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017904.545062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017904.545065:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017904.545068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334363904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017904.545074:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334364032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017904.545087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017904.545093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017904/real 1734017904] req@ffff8800a78a1f80 x1818247334364032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017904.545102:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334364032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017904.545110:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017904.545112:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017904.545115:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334364032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017904.545120:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334364160:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017904.545132:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017904.545138:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017904/real 1734017904] req@ffff8800a78a0a80 x1818247334364160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017904.545187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334364160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017904.545196:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017904.545198:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017904.545202:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334364160:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017904.545208:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334364288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017904.545228:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017904.545235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017904/real 1734017904] req@ffff8800a78a2680 x1818247334364288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017904.545246:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334364288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017904.545253:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017904.545256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017904.545259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334364288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017904.545264:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334364416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017904.545276:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017904.545282:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017904/real 1734017904] req@ffff8800a78a0700 x1818247334364416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017904.545292:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334364416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017904.545299:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017904.545301:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017904.545304:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334364416:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017905.154686:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017906.156736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017907.158736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017908.160738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017909.162768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017909.552615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017909.552627:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017909.552632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017909.552646:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2176 00000100:00080000:2.0:1734017909.552658:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017909.552668:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017909.552717:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334364544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017909.552763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017909.552767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017909.552769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017909.552774:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2176 00000100:00080000:2.0:1734017909.552778:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017909.552782:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017909.552792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017909.552805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334364672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017909.552814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017909.552817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017909.552819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017909.552823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2176 00000100:00080000:2.0:1734017909.552826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017909.552830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017909.552835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017909.552844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334364800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017909.552853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017909.552856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017909.552858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017909.552861:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2176 00000100:00080000:2.0:1734017909.552864:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017909.552868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017909.552872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017909.552881:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334364928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017909.552891:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017909.552894:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017909.552896:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017909.552900:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2176 00000100:00100000:1.0:1734017909.552901:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334364544:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017909.552902:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017909.552906:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017909.552910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017909.552918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334365056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017909.552926:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017909.552929:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017909.552930:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017909.552934:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2176 00000100:00080000:2.0:1734017909.552937:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017909.552940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017909.552944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017909.552953:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334365184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017909.552961:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017909.552965:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017909.552967:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017909.552970:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2176 00000100:00080000:2.0:1734017909.552974:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017909.552975:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017909.552978:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017909.552982:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017909.552990:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334365312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017909.552998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017909/real 1734017909] req@ffff8800a78a0700 x1818247334364544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017909.553026:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334364544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017909.553037:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017909.553041:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017909.553046:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334364544:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017909.553054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334364672:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017909.553119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017909.553128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017909/real 1734017909] req@ffff8800a78a2680 x1818247334364672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017909.553140:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334364672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017909.553148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017909.553151:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017909.553154:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334364672:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017909.553161:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334364800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017909.553176:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017909.553182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017909/real 1734017909] req@ffff8800a78a0a80 x1818247334364800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017909.553193:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334364800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017909.553200:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017909.553203:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017909.553207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334364800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017909.553234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334364928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017909.553248:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017909.553254:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017909/real 1734017909] req@ffff8800a78a1f80 x1818247334364928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017909.553264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334364928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017909.553272:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017909.553274:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017909.553277:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334364928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017909.553283:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334365056:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017909.553296:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017909.553302:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017909/real 1734017909] req@ffff8800a78a3480 x1818247334365056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017909.553312:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334365056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017909.553320:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017909.553322:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017909.553325:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334365056:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017909.553331:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334365184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017909.553347:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017909.553353:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017909/real 1734017909] req@ffff8800a78a3b80 x1818247334365184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017909.553364:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334365184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017909.553371:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017909.553373:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017909.553376:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334365184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017909.553382:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334365312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017909.553394:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017909.553400:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017909/real 1734017909] req@ffff8800a78a1880 x1818247334365312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017909.553410:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334365312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017909.553417:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017909.553419:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017909.553422:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334365312:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017910.164770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017911.166730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017912.168753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017913.170755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017914.172767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017914.560599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017914.560608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017914.560613:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017914.560623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2181 00000100:00080000:2.0:1734017914.560632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017914.560640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017914.560678:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334365440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017914.560713:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017914.560718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017914.560720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017914.560724:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2181 00000100:00080000:2.0:1734017914.560727:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017914.560731:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017914.560741:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017914.560753:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334365568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017914.560764:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017914.560767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017914.560768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017914.560772:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2181 00000100:00080000:2.0:1734017914.560775:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017914.560779:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017914.560783:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017914.560792:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334365696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017914.560801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017914.560804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017914.560806:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017914.560809:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2181 00000100:00080000:2.0:1734017914.560811:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017914.560815:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017914.560819:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017914.560820:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334365440:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017914.560827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334365824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017914.560836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017914.560839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017914.560841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017914.560844:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2181 00000100:00080000:2.0:1734017914.560847:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017914.560850:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017914.560854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017914.560861:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334365952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017914.560869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017914.560872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017914.560873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017914.560877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2181 00000100:00080000:2.0:1734017914.560880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017914.560883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017914.560886:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017914.560887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017914.560894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334366080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017914.560902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017914.560905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017914.560906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017914.560908:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017914/real 1734017914] req@ffff8800a78a1880 x1818247334365440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017914.560910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2181 00000100:00080000:2.0:1734017914.560913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017914.560916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017914.560920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017914.560927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334366208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017914.560934:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334365440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017914.560944:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017914.560947:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017914.560952:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334365440:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017914.560959:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334365568:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017914.560975:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017914.561027:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017914/real 1734017914] req@ffff8800a78a3b80 x1818247334365568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017914.561039:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334365568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017914.561047:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017914.561050:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017914.561053:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334365568:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017914.561059:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334365696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017914.561077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017914.561083:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017914/real 1734017914] req@ffff8800a78a3480 x1818247334365696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017914.561094:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334365696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017914.561102:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017914.561105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017914.561108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334365696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017914.561133:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334365824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017914.561146:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017914.561153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017914/real 1734017914] req@ffff8800a78a1f80 x1818247334365824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017914.561163:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334365824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017914.561170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017914.561173:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017914.561175:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334365824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017914.561181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334365952:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017914.561193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017914.561199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017914/real 1734017914] req@ffff8800a78a0a80 x1818247334365952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017914.561209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334365952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017914.561216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017914.561218:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017914.561221:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334365952:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017914.561227:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334366080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017914.561242:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017914.561249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017914/real 1734017914] req@ffff8800a78a2680 x1818247334366080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017914.561259:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334366080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017914.561266:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017914.561268:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017914.561271:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334366080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017914.561276:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334366208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017914.561288:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017914.561294:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017914/real 1734017914] req@ffff8800a78a0700 x1818247334366208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017914.561304:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334366208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017914.561311:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017914.561313:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017914.561316:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334366208:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017915.174738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017916.176725:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017917.178752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017918.180737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017919.182738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017919.568607:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017919.568617:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017919.568622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017919.568632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2186 00000100:00080000:2.0:1734017919.568642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017919.568649:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017919.568687:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334366336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017919.568721:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017919.568725:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017919.568727:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017919.568732:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2186 00000100:00080000:2.0:1734017919.568735:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017919.568739:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017919.568749:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017919.568761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334366464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017919.568771:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017919.568774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017919.568775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017919.568780:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2186 00000100:00080000:2.0:1734017919.568783:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017919.568786:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017919.568791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017919.568800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334366592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017919.568809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017919.568812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017919.568813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017919.568817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2186 00000100:00080000:2.0:1734017919.568819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017919.568822:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017919.568826:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017919.568833:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334366336:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017919.568834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334366720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017919.568843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017919.568846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017919.568847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017919.568850:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2186 00000100:00080000:2.0:1734017919.568853:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017919.568856:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017919.568860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017919.568867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334366848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017919.568875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017919.568878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017919.568880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017919.568884:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2186 00000100:00080000:2.0:1734017919.568889:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017919.568931:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017919.568931:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017919.568937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017919.568946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334366976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017919.568954:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017919/real 1734017919] req@ffff8800a78a0700 x1818247334366336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017919.568955:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017919.568958:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017919.568960:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017919.568963:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2186 00000100:00080000:2.0:1734017919.568967:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017919.568970:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017919.568974:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017919.568982:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334367104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017919.568983:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334366336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017919.568993:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017919.568996:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017919.569001:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334366336:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017919.569009:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334366464:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017919.569029:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017919.569036:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017919/real 1734017919] req@ffff8800a78a2680 x1818247334366464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017919.569047:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334366464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017919.569055:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017919.569057:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017919.569060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334366464:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017919.569066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334366592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017919.569080:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017919.569086:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017919/real 1734017919] req@ffff8800a78a0a80 x1818247334366592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017919.569096:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334366592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017919.569104:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017919.569106:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017919.569109:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334366592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017919.569134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334366720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017919.569147:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017919.569153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017919/real 1734017919] req@ffff8800a78a1f80 x1818247334366720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017919.569164:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334366720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017919.569171:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017919.569173:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017919.569176:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334366720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017919.569181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334366848:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017919.569193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017919.569199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017919/real 1734017919] req@ffff8800a78a3480 x1818247334366848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017919.569209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334366848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017919.569216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017919.569218:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017919.569221:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334366848:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017919.569227:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334366976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017919.569242:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017919.569248:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017919/real 1734017919] req@ffff8800a78a3b80 x1818247334366976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017919.569258:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334366976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017919.569265:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017919.569267:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017919.569270:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334366976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017919.569275:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334367104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017919.569287:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017919.569293:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017919/real 1734017919] req@ffff8800a78a1880 x1818247334367104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017919.569303:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334367104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017919.569310:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017919.569312:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017919.569315:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334367104:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017920.184752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017921.186720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017922.188757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017923.190755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017924.192767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017924.576610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017924.576624:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017924.576629:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017924.576643:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2191 00000100:00080000:2.0:1734017924.576655:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017924.576665:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017924.576715:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334367232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017924.576761:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017924.576765:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017924.576767:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017924.576772:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2191 00000100:00080000:2.0:1734017924.576776:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017924.576780:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017924.576790:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017924.576803:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334367360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017924.576850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017924.576853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017924.576855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017924.576860:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2191 00000100:00080000:2.0:1734017924.576863:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017924.576867:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017924.576872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017924.576883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334367488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017924.576893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017924.576897:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017924.576899:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017924.576903:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2191 00000100:00080000:2.0:1734017924.576905:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017924.576906:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334367232:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017924.576909:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017924.576914:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017924.576924:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334367616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017924.576933:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017924.576936:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017924.576937:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017924.576941:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2191 00000100:00080000:2.0:1734017924.576943:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017924.576947:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017924.576951:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017924.576959:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334367744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017924.576967:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017924.576970:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017924.576972:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017924.576976:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2191 00000100:00080000:2.0:1734017924.576979:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017924.576983:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017924.576987:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017924.576995:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334367872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017924.577003:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017924.577004:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334367360:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017924.577007:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017924.577009:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017924.577013:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2191 00000100:00080000:2.0:1734017924.577016:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017924.577019:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017924.577023:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017924.577032:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334368000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017924.577034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334367488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017924.577052:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334367616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017924.577068:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334367744:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017924.577084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334367872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017924.577104:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334368000:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017925.194732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017926.196731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017927.198760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017927.582793:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:0.0:1734017927.582839:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017927.582858:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017927.582896:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017927.582903:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017927.582910:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017927.582917:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017927.582923:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:0.0:1734017927.582929:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734017927.582969:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017924/real 1734017927] req@ffff8800a78a0700 x1818247334368000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017927.583007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334368000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017927.583025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017927.583031:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017927.583039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334368000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017927.583051:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017924/real 1734017927] req@ffff8800a78a3480 x1818247334367488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017927.583070:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334367488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017927.583082:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017927.583086:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017927.583090:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334367488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017927.583100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017924/real 1734017927] req@ffff8800a78a1f80 x1818247334367616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017927.583117:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334367616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017927.583129:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017927.583132:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017927.583137:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334367616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017927.583146:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017924/real 1734017927] req@ffff8800a78a0a80 x1818247334367744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017927.583165:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334367744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017927.583177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017927.583181:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017927.583186:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334367744:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017927.583196:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017924/real 1734017927] req@ffff8800a78a2680 x1818247334367872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017927.583214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334367872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017927.583226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017927.583230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017927.583235:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334367872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017927.583245:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017924/real 1734017927] req@ffff8800a78a1880 x1818247334367232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017927.583264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334367232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017927.583286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017927.583289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017927.583294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334367232:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017927.583304:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017924/real 1734017927] req@ffff8800a78a3b80 x1818247334367360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017927.583321:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334367360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017927.583332:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017927.583336:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017927.583341:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334367360:192.168.201.116@tcp:38:kworker.0 00010000:00010000:3.0:1734017928.200724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017929.202759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017929.584614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017929.584623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017929.584628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017929.584640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2196 00000100:00080000:2.0:1734017929.584649:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017929.584657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017929.584695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334368128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017929.584770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017929.584775:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017929.584777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017929.584782:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2196 00000100:00080000:2.0:1734017929.584786:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017929.584790:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017929.584800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017929.584813:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334368256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017929.584825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017929.584828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017929.584829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017929.584834:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2196 00000100:00080000:2.0:1734017929.584836:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017929.584840:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017929.584845:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334368128:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017929.584845:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017929.584854:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334368384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017929.584863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017929.584866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017929.584868:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017929.584871:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2196 00000100:00080000:2.0:1734017929.584874:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017929.584877:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017929.584881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017929.584889:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334368512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017929.584897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017929.584900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017929.584901:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017929.584905:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2196 00000100:00080000:2.0:1734017929.584908:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017929.584911:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017929.584912:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017929.584916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017929.584923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334368640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017929.584931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017929/real 1734017929] req@ffff8800a78a3b80 x1818247334368128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017929.584931:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017929.584934:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017929.584935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017929.584938:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2196 00000100:00080000:2.0:1734017929.584941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017929.584944:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017929.584948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017929.584951:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334368128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017929.584956:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334368768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017929.584962:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017929.584963:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734017929.584965:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734017929.584966:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017929.584968:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017929.584970:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334368128:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017929.584971:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2196 00000100:00080000:2.0:1734017929.584974:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017929.584977:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017929.584978:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334368256:192.168.201.116@tcp:38:kworker.0 00000008:00100000:2.0:1734017929.584981:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017929.584988:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334368896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017929.584995:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017929.585002:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017929/real 1734017929] req@ffff8800a78a1880 x1818247334368256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017929.585014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334368256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017929.585021:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017929.585024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017929.585027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334368256:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017929.585049:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334368384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017929.585063:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017929.585069:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017929/real 1734017929] req@ffff8800a78a2680 x1818247334368384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017929.585079:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334368384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017929.585087:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017929.585089:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017929.585092:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334368384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017929.585098:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334368512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017929.585110:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017929.585116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017929/real 1734017929] req@ffff8800a78a0a80 x1818247334368512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017929.585126:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334368512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017929.585140:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017929.585142:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017929.585145:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334368512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017929.585150:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334368640:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017929.585163:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017929.585169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017929/real 1734017929] req@ffff8800a78a1f80 x1818247334368640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017929.585180:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334368640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017929.585187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017929.585189:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017929.585192:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334368640:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017929.585197:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334368768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017929.585209:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017929.585215:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017929/real 1734017929] req@ffff8800a78a3480 x1818247334368768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017929.585226:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334368768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017929.585233:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017929.585235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017929.585238:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334368768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017929.585244:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334368896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017929.585257:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017929.585263:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017929/real 1734017929] req@ffff8800a78a0700 x1818247334368896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017929.585273:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334368896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017929.585280:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017929.585282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017929.585285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334368896:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017930.204789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017931.206729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017932.208766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017933.210749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017934.212756:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017934.592599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017934.592608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017934.592613:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017934.592623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2201 00000100:00080000:2.0:1734017934.592632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017934.592679:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017934.592719:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334369024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017934.592753:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017934.592757:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017934.592759:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017934.592763:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2201 00000100:00080000:2.0:1734017934.592767:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017934.592771:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017934.592781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017934.592793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334369152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017934.592803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017934.592807:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017934.592808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017934.592813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2201 00000100:00080000:2.0:1734017934.592815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017934.592819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017934.592825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017934.592834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334369280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017934.592843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017934.592846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017934.592848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017934.592851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2201 00000100:00080000:2.0:1734017934.592854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017934.592857:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017934.592860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734017934.592869:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334369024:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017934.592869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334369408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017934.592878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017934.592881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017934.592882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017934.592885:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2201 00000100:00080000:2.0:1734017934.592888:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017934.592891:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017934.592895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017934.592902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334369536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017934.592911:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017934.592914:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017934.592915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017934.592919:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2201 00000100:00080000:2.0:1734017934.592924:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017934.592927:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017934.592931:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017934.592935:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017934.592939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334369664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017934.592947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017934.592950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017934.592952:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017934.592955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2201 00000100:00100000:1.0:1734017934.592956:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017934/real 1734017934] req@ffff8800a78a0700 x1818247334369024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017934.592958:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017934.592961:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017934.592965:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017934.592972:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334369792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017934.592976:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334369024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017934.592986:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017934.592989:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017934.592994:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334369024:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017934.593001:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334369152:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017934.593018:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017934.593025:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017934/real 1734017934] req@ffff8800a78a3480 x1818247334369152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017934.593044:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334369152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017934.593052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017934.593055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017934.593058:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334369152:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017934.593064:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334369280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017934.593077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017934.593083:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017934/real 1734017934] req@ffff8800a78a1f80 x1818247334369280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017934.593093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334369280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017934.593100:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017934.593103:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017934.593106:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334369280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017934.593130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334369408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017934.593143:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017934.593149:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017934/real 1734017934] req@ffff8800a78a0a80 x1818247334369408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017934.593159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334369408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017934.593167:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017934.593169:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017934.593172:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334369408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017934.593178:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334369536:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017934.593191:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017934.593197:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017934/real 1734017934] req@ffff8800a78a2680 x1818247334369536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017934.593207:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334369536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017934.593215:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017934.593217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017934.593220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334369536:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017934.593225:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334369664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017934.593236:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017934.593242:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017934/real 1734017934] req@ffff8800a78a1880 x1818247334369664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017934.593252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334369664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017934.593259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017934.593261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017934.593264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334369664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017934.593270:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334369792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017934.593285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017934.593291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017934/real 1734017934] req@ffff8800a78a3b80 x1818247334369792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017934.593301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334369792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017934.593308:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017934.593310:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017934.593313:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334369792:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017935.214782:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017936.216816:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017937.218788:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017938.220764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017939.222782:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017939.600623:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017939.600633:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017939.600637:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017939.600647:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2206 00000100:00080000:2.0:1734017939.600657:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017939.600664:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017939.600701:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334369920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017939.600746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017939.600752:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017939.600755:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017939.600761:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2206 00000100:00080000:2.0:1734017939.600773:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017939.600779:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017939.600793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017939.600812:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334370048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017939.600826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017939.600827:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334369920:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017939.600831:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017939.600834:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017939.600839:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2206 00000100:00080000:2.0:1734017939.600843:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017939.600848:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017939.600854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017939.600867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334370176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017939.600879:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017939.600883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017939.600886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017939.600890:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2206 00000100:00080000:2.0:1734017939.600893:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017939.600899:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017939.600899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017939.600905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017939.600916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334370304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017939.600922:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017939/real 1734017939] req@ffff8800a78a3b80 x1818247334369920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017939.600930:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017939.600935:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017939.600938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017939.600942:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2206 00000100:00100000:1.0:1734017939.600945:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334369920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017939.600947:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017939.600952:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734017939.600959:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1734017939.600959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734017939.600964:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017939.600971:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334369920:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734017939.600971:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334370432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017939.600984:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017939.600988:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017939.600991:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017939.600996:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2206 00000100:00100000:1.0:1734017939.600997:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334370048:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017939.601000:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017939.601006:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017939.601012:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017939.601023:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017939.601024:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334370560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017939.601034:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017939/real 1734017939] req@ffff8800a78a1880 x1818247334370048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017939.601038:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017939.601044:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017939.601046:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017939.601051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334370048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017939.601051:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2206 00000100:00080000:2.0:1734017939.601056:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017939.601061:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734017939.601065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1734017939.601067:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017939.601068:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017939.601074:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334370048:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734017939.601078:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334370688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017939.601083:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334370176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017939.601107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017939.601118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017939/real 1734017939] req@ffff8800a78a2680 x1818247334370176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017939.601136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334370176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017939.601149:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017939.601153:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017939.601158:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334370176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017939.601167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334370304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017939.601192:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017939.601202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017939/real 1734017939] req@ffff8800a78a0a80 x1818247334370304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017939.601213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334370304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017939.601220:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017939.601223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017939.601226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334370304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017939.601232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334370432:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017939.601247:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017939.601254:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017939/real 1734017939] req@ffff8800a78a1f80 x1818247334370432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017939.601264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334370432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017939.601272:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017939.601274:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017939.601277:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334370432:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017939.601294:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334370560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017939.601307:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017939.601313:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017939/real 1734017939] req@ffff8800a78a3480 x1818247334370560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017939.601323:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334370560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017939.601330:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017939.601333:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017939.601336:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334370560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017939.601341:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334370688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017939.601353:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017939.601359:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017939/real 1734017939] req@ffff8800a78a0700 x1818247334370688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017939.601370:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334370688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017939.601378:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017939.601380:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017939.601383:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334370688:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017940.224768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017941.226803:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017942.228760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017943.230759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017944.232776:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017944.608610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017944.608619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017944.608624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017944.608635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2211 00000100:00080000:2.0:1734017944.608644:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017944.608652:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017944.608689:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334370816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017944.608723:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017944.608727:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017944.608729:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017944.608733:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2211 00000100:00080000:2.0:1734017944.608736:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017944.608740:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017944.608751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017944.608762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334370944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017944.608773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017944.608776:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017944.608778:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017944.608782:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2211 00000100:00080000:2.0:1734017944.608785:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017944.608788:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017944.608793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017944.608802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334371072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017944.608810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017944.608813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017944.608815:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017944.608818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2211 00000100:00080000:2.0:1734017944.608820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017944.608824:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017944.608828:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334370816:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017944.608828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017944.608835:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334371200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017944.608844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017944.608847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017944.608848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017944.608852:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2211 00000100:00080000:2.0:1734017944.608855:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017944.608858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017944.608862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017944.608870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334371328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017944.608877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017944.608880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017944.608882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017944.608885:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2211 00000100:00080000:2.0:1734017944.608888:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017944.608891:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017944.608894:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017944.608895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017944.608902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334371456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017944.608910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017944.608913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017944.608914:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017944.608916:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017944/real 1734017944] req@ffff8800a78a0700 x1818247334370816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017944.608918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2211 00000100:00080000:2.0:1734017944.608920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017944.608923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017944.608927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017944.608934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334371584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017944.608936:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334370816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017944.608946:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017944.608950:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017944.608955:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334370816:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017944.608962:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334370944:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017944.608979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017944.608986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017944/real 1734017944] req@ffff8800a78a3480 x1818247334370944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017944.608997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334370944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017944.609005:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017944.609008:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017944.609011:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334370944:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017944.609018:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334371072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017944.609032:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017944.609037:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017944/real 1734017944] req@ffff8800a78a1f80 x1818247334371072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017944.609047:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334371072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017944.609055:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017944.609057:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017944.609061:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334371072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017944.609087:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334371200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017944.609100:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017944.609106:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017944/real 1734017944] req@ffff8800a78a0a80 x1818247334371200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017944.609116:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334371200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017944.609124:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017944.609127:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017944.609130:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334371200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017944.609135:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334371328:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017944.609150:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017944.609156:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017944/real 1734017944] req@ffff8800a78a2680 x1818247334371328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017944.609166:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334371328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017944.609173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017944.609175:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017944.609178:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334371328:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017944.609183:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334371456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017944.609195:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017944.609201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017944/real 1734017944] req@ffff8800a78a1880 x1818247334371456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017944.609210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334371456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017944.609217:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017944.609220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017944.609223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334371456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017944.609228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334371584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017944.609240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017944.609246:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017944/real 1734017944] req@ffff8800a78a3b80 x1818247334371584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017944.609262:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334371584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017944.609269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017944.609271:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017944.609274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334371584:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017945.234756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017946.236713:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017947.238767:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017948.240835:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017949.242764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017949.616603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017949.616616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017949.616622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017949.616634:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2216 00000100:00080000:2.0:1734017949.616646:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017949.616655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017949.616711:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334371712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017949.616761:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017949.616766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017949.616769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017949.616776:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2216 00000100:00080000:2.0:1734017949.616782:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017949.616788:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017949.616802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017949.616824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334371840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017949.616840:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017949.616845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017949.616847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017949.616855:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2216 00000100:00080000:2.0:1734017949.616860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017949.616866:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017949.616874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017949.616891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334371968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017949.616897:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334371712:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017949.616908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017949.616913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017949.616916:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017949.616922:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2216 00000100:00080000:2.0:1734017949.616926:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017949.616932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017949.616939:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017949.616950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334372096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017949.616960:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017949.616964:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017949.616965:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017949.616969:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2216 00000800:00000100:1.0:1734017949.616970:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734017949.616973:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017949.616976:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017949.616981:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017949.616989:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334372224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017949.616994:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017949/real 1734017949] req@ffff8800a78a3b80 x1818247334371712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017949.616998:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017949.617001:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017949.617003:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017949.617007:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2216 00000100:00080000:2.0:1734017949.617010:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017949.617014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334371712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017949.617014:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017949.617019:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017949.617026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1734017949.617027:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334372352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017949.617030:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017949.617035:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334371712:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017949.617035:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017949.617038:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017949.617040:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017949.617043:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334371840:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734017949.617043:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2216 00000100:00080000:2.0:1734017949.617046:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017949.617049:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017949.617054:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017949.617061:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017949.617062:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334372480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017949.617068:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017949/real 1734017949] req@ffff8800a78a1880 x1818247334371840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017949.617081:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334371840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017949.617089:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017949.617093:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017949.617096:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334371840:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017949.617123:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334371968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017949.617137:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017949.617144:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017949/real 1734017949] req@ffff8800a78a2680 x1818247334371968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017949.617154:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334371968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017949.617162:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017949.617164:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017949.617168:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334371968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017949.617175:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334372096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017949.617188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017949.617194:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017949/real 1734017949] req@ffff8800a78a0a80 x1818247334372096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017949.617204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334372096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017949.617211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017949.617213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017949.617216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334372096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017949.617223:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334372224:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017949.617239:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017949.617246:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017949/real 1734017949] req@ffff8800a78a1f80 x1818247334372224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017949.617256:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334372224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017949.617264:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017949.617266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017949.617269:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334372224:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017949.617275:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334372352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017949.617288:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017949.617294:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017949/real 1734017949] req@ffff8800a78a3480 x1818247334372352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017949.617304:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334372352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017949.617311:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017949.617313:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017949.617316:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334372352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017949.617322:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334372480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017949.617335:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017949.617341:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017949/real 1734017949] req@ffff8800a78a0700 x1818247334372480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017949.617358:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334372480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017949.617365:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017949.617368:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017949.617371:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334372480:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017950.244740:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017951.246736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017952.248699:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017953.250757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017954.252597:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017954.624623:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017954.624633:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017954.624637:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017954.624648:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2221 00000100:00080000:2.0:1734017954.624656:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017954.624664:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017954.624702:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334372608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017954.624745:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017954.624749:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017954.624751:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017954.624756:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2221 00000100:00080000:2.0:1734017954.624759:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017954.624763:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017954.624773:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017954.624785:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334372736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017954.624802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017954.624805:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017954.624807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017954.624811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2221 00000100:00080000:2.0:1734017954.624814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017954.624818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017954.624823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017954.624831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334372864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017954.624840:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017954.624843:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017954.624844:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017954.624848:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2221 00000100:00080000:2.0:1734017954.624850:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017954.624854:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017954.624858:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334372608:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734017954.624859:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017954.624867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334372992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017954.624876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017954.624879:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017954.624880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017954.624883:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2221 00000100:00080000:2.0:1734017954.624886:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017954.624889:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017954.624893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017954.624900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334373120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017954.624908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017954.624911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017954.624913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017954.624916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2221 00000100:00080000:2.0:1734017954.624918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017954.624922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734017954.624926:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734017954.624926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017954.624934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334373248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017954.624942:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017954.624945:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017954.624947:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017954/real 1734017954] req@ffff8800a78a0700 x1818247334372608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017954.624947:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017954.624950:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2221 00000100:00080000:2.0:1734017954.624953:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017954.624957:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017954.624960:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017954.624967:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334372608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017954.624968:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334373376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017954.624978:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017954.624981:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017954.624986:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334372608:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017954.624993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334372736:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017954.625010:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017954.625016:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017954/real 1734017954] req@ffff8800a78a3480 x1818247334372736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017954.625027:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334372736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017954.625035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017954.625039:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017954.625042:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334372736:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017954.625048:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334372864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017954.625062:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017954.625068:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017954/real 1734017954] req@ffff8800a78a1f80 x1818247334372864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017954.625078:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334372864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017954.625086:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017954.625088:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017954.625092:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334372864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017954.625120:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334372992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017954.625133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017954.625139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017954/real 1734017954] req@ffff8800a78a0a80 x1818247334372992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017954.625149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334372992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017954.625157:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017954.625160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017954.625162:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334372992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017954.625168:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334373120:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017954.625184:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017954.625190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017954/real 1734017954] req@ffff8800a78a2680 x1818247334373120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017954.625200:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334373120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017954.625207:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017954.625209:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017954.625212:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334373120:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017954.625218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334373248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017954.625229:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017954.625235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017954/real 1734017954] req@ffff8800a78a1880 x1818247334373248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017954.625245:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334373248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017954.625252:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017954.625254:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017954.625257:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334373248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017954.625262:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334373376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017954.625274:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017954.625280:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017954/real 1734017954] req@ffff8800a78a3b80 x1818247334373376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017954.625338:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334373376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017954.625346:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017954.625349:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017954.625352:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334373376:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017955.254687:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017956.256739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017957.258770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017958.260766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017959.262775:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017959.632594:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017959.632605:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017959.632610:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017959.632620:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2226 00000100:00080000:2.0:1734017959.632629:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017959.632636:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017959.632676:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334373504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017959.632713:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017959.632717:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017959.632719:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017959.632723:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2226 00000100:00080000:2.0:1734017959.632727:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017959.632730:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017959.632741:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017959.632752:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334373632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017959.632763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017959.632766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017959.632768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017959.632772:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2226 00000100:00080000:2.0:1734017959.632774:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017959.632778:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017959.632782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017959.632791:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334373760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017959.632800:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017959.632803:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017959.632804:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017959.632807:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2226 00000100:00080000:2.0:1734017959.632810:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017959.632813:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017959.632817:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017959.632825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334373888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017959.632835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017959.632837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017959.632839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017959.632843:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2226 00000100:00080000:2.0:1734017959.632846:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017959.632848:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334373504:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017959.632849:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017959.632853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017959.632861:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334374016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017959.632869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017959.632871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017959.632873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017959.632876:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2226 00000100:00080000:2.0:1734017959.632879:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017959.632882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017959.632886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017959.632893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334374144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017959.632900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017959.632903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017959.632905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017959.632908:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2226 00000100:00080000:2.0:1734017959.632911:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017959.632914:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017959.632917:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017959.632925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334374272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017959.632938:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017959.632966:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017959/real 1734017959] req@ffff8800a78a3b80 x1818247334373504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017959.632986:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334373504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017959.632999:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017959.633003:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017959.633009:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334373504:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017959.633017:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334373632:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017959.633036:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017959.633043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017959/real 1734017959] req@ffff8800a78a1880 x1818247334373632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017959.633054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334373632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017959.633063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017959.633067:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017959.633070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334373632:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017959.633078:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334373760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017959.633093:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017959.633100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017959/real 1734017959] req@ffff8800a78a2680 x1818247334373760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017959.633110:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334373760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017959.633118:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017959.633121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017959.633124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334373760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017959.633154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334373888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017959.633168:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017959.633174:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017959/real 1734017959] req@ffff8800a78a0a80 x1818247334373888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017959.633185:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334373888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017959.633193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017959.633196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017959.633198:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334373888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017959.633205:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334374016:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017959.633265:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017959.633273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017959/real 1734017959] req@ffff8800a78a1f80 x1818247334374016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017959.633285:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334374016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017959.633293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017959.633296:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017959.633299:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334374016:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017959.633305:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334374144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017959.633317:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017959.633324:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017959/real 1734017959] req@ffff8800a78a3480 x1818247334374144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017959.633334:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334374144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017959.633342:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017959.633344:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017959.633347:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334374144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017959.633353:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334374272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017959.633367:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017959.633373:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017959/real 1734017959] req@ffff8800a78a0700 x1818247334374272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017959.633390:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334374272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017959.633398:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017959.633401:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017959.633404:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334374272:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017960.264759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017961.266747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017962.268781:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017963.270805:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017964.272760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017964.640622:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017964.640631:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017964.640636:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017964.640648:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2231 00000100:00080000:2.0:1734017964.640656:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017964.640665:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017964.640704:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334374400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017964.640740:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017964.640744:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017964.640746:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017964.640751:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2231 00000100:00080000:2.0:1734017964.640754:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017964.640758:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017964.640768:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017964.640780:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334374528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017964.640790:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017964.640793:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017964.640795:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017964.640800:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2231 00000100:00080000:2.0:1734017964.640803:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017964.640806:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017964.640812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017964.640822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334374656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017964.640831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017964.640834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017964.640836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017964.640839:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2231 00000100:00080000:2.0:1734017964.640842:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017964.640845:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017964.640848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017964.640857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334374784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017964.640866:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334374400:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017964.640866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017964.640869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017964.640871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017964.640874:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2231 00000100:00080000:2.0:1734017964.640877:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017964.640880:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017964.640884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017964.640892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334374912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017964.640900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017964.640903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017964.640905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017964.640909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2231 00000100:00080000:2.0:1734017964.640911:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017964.640915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017964.640919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017964.640926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334375040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017964.640934:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017964.640937:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017964.640938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017964.640941:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2231 00000100:00080000:2.0:1734017964.640945:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017964.640948:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017964.640951:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017964.640957:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734017964.640959:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334375168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017964.640985:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017964/real 1734017964] req@ffff8800a78a0700 x1818247334374400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017964.641006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334374400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017964.641019:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017964.641023:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017964.641028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334374400:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017964.641036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334374528:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017964.641055:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017964.641062:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017964/real 1734017964] req@ffff8800a78a3480 x1818247334374528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017964.641073:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334374528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017964.641082:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017964.641086:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017964.641089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334374528:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017964.641097:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334374656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017964.641112:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017964.641118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017964/real 1734017964] req@ffff8800a78a1f80 x1818247334374656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017964.641172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334374656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017964.641181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017964.641184:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017964.641188:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334374656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017964.641219:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334374784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017964.641237:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017964.641244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017964/real 1734017964] req@ffff8800a78a0a80 x1818247334374784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017964.641255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334374784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017964.641263:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017964.641265:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017964.641268:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334374784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017964.641275:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334374912:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017964.641292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017964.641298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017964/real 1734017964] req@ffff8800a78a2680 x1818247334374912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017964.641309:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334374912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017964.641317:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017964.641319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017964.641322:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334374912:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017964.641328:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334375040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017964.641340:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017964.641346:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017964/real 1734017964] req@ffff8800a78a1880 x1818247334375040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017964.641356:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334375040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017964.641364:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017964.641367:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017964.641370:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334375040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017964.641376:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334375168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017964.641389:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017964.641395:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017964/real 1734017964] req@ffff8800a78a3b80 x1818247334375168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017964.641411:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334375168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017964.641419:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017964.641421:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017964.641424:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334375168:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017965.274709:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017966.276761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017967.278748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017968.280767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017969.282758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734017969.648609:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017969.648618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734017969.648622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017969.648633:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2236 00000100:00080000:2.0:1734017969.648641:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017969.648649:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734017969.648688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334375296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017969.648730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017969.648735:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017969.648736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017969.648741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2236 00000100:00080000:2.0:1734017969.648744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017969.648748:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017969.648758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017969.648770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334375424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017969.648779:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017969.648782:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017969.648784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017969.648788:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2236 00000100:00080000:2.0:1734017969.648790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017969.648794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017969.648799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734017969.648808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334375552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017969.648816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017969.648820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017969.648821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017969.648825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2236 00000100:00080000:2.0:1734017969.648827:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017969.648831:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017969.648835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734017969.648843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334375680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017969.648853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017969.648855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734017969.648857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017969.648860:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2236 00000100:00080000:2.0:1734017969.648862:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017969.648865:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017969.648869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017969.648877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334375808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017969.648878:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334375296:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734017969.648885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017969.648888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017969.648890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017969.648894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2236 00000100:00080000:2.0:1734017969.648896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017969.648899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017969.648903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017969.648911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334375936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017969.648918:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017969.648922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734017969.648923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017969.648927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2236 00000100:00080000:2.0:1734017969.648930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734017969.648933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734017969.648937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017969.648944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334376064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017969.648969:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017969.648996:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017969/real 1734017969] req@ffff8800a78a3b80 x1818247334375296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017969.649019:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334375296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017969.649031:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017969.649035:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017969.649072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334375296:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017969.649081:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334375424:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017969.649104:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017969.649112:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017969/real 1734017969] req@ffff8800a78a1880 x1818247334375424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017969.649123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334375424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017969.649132:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017969.649136:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017969.649140:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334375424:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017969.649147:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334375552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017969.649162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017969.649169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017969/real 1734017969] req@ffff8800a78a2680 x1818247334375552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017969.649179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334375552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017969.649187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017969.649190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017969.649194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334375552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017969.649224:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334375680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017969.649238:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017969.649244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017969/real 1734017969] req@ffff8800a78a0a80 x1818247334375680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017969.649255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334375680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017969.649263:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017969.649266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017969.649269:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334375680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017969.649275:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334375808:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017969.649292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017969.649299:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017969/real 1734017969] req@ffff8800a78a1f80 x1818247334375808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017969.649309:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334375808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017969.649317:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017969.649319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017969.649322:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334375808:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017969.649328:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334375936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017969.649340:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017969.649346:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017969/real 1734017969] req@ffff8800a78a3480 x1818247334375936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017969.649356:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334375936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017969.649363:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017969.649366:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017969.649369:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334375936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017969.649374:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334376064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017969.649388:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017969.649394:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017969/real 1734017969] req@ffff8800a78a0700 x1818247334376064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017969.649410:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334376064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017969.649419:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017969.649421:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017969.649424:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334376064:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017970.284711:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017971.286785:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017972.288742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017973.290748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017974.292778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017974.656686:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017974.656695:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017974.656699:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017974.656710:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2241 00000100:00080000:0.0:1734017974.656718:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017974.656725:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017974.656762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334376192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017974.656800:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017974.656804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017974.656806:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017974.656811:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2241 00000100:00080000:0.0:1734017974.656814:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017974.656818:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017974.656828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017974.656840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334376320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017974.656850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017974.656854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017974.656856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017974.656860:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2241 00000100:00080000:0.0:1734017974.656863:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017974.656866:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017974.656871:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017974.656879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334376448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017974.656888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017974.656892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017974.656893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017974.656897:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2241 00000100:00080000:0.0:1734017974.656899:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017974.656902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017974.656903:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334376192:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734017974.656906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017974.656916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334376576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017974.656925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017974.656928:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017974.656930:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017974.656935:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2241 00000100:00080000:0.0:1734017974.656938:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017974.656941:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017974.656945:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017974.656952:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334376704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017974.657002:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734017974.657003:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734017974.657005:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017974.657006:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017974.657011:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2241 00000100:00080000:0.0:1734017974.657014:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017974.657018:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017974.657022:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017974.657026:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017974/real 1734017974] req@ffff8800a934ea00 x1818247334376192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017974.657032:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334376832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017974.657040:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017974.657044:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017974.657045:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017974.657046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334376192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017974.657049:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2241 00000100:00080000:0.0:1734017974.657052:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017974.657055:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734017974.657056:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1734017974.657058:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017974.657060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017974.657065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334376192:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734017974.657066:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334376960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017974.657073:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334376320:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017974.657092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017974.657099:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017974/real 1734017974] req@ffff8800a934c380 x1818247334376320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017974.657110:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334376320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017974.657118:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017974.657122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017974.657125:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334376320:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017974.657148:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334376448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017974.657162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017974.657168:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017974/real 1734017974] req@ffff8800a934ca80 x1818247334376448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017974.657179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334376448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017974.657187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017974.657189:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017974.657192:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334376448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017974.657198:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334376576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017974.657210:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017974.657215:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017974/real 1734017974] req@ffff8800a934ed80 x1818247334376576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017974.657225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334376576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017974.657232:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017974.657235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017974.657238:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334376576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017974.657243:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334376704:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017974.657259:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017974.657265:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017974/real 1734017974] req@ffff8800a934c700 x1818247334376704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017974.657276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334376704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017974.657284:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017974.657286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017974.657289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334376704:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017974.657294:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334376832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017974.657306:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017974.657312:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017974/real 1734017974] req@ffff8800a934d880 x1818247334376832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017974.657322:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334376832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017974.657329:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017974.657331:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017974.657334:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334376832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017974.657340:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334376960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017974.657352:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017974.657358:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017974/real 1734017974] req@ffff8800a934dc00 x1818247334376960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017974.657375:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334376960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017974.657382:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017974.657384:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017974.657387:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334376960:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017975.294769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017976.296701:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017977.298795:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017978.300746:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017979.302721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017979.664610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017979.664619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017979.664624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017979.664633:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2246 00000100:00080000:0.0:1734017979.664642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017979.664650:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017979.664685:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334377088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017979.664718:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017979.664722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017979.664725:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017979.664729:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2246 00000100:00080000:0.0:1734017979.664732:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017979.664736:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017979.664746:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017979.664757:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334377216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017979.664767:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017979.664770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017979.664772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017979.664776:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2246 00000100:00080000:0.0:1734017979.664779:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017979.664783:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017979.664789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017979.664799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334377344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017979.664809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017979.664812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017979.664814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017979.664817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2246 00000100:00100000:1.0:1734017979.664818:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334377088:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734017979.664820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017979.664823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017979.664827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017979.664835:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334377472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017979.664844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017979.664847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017979.664849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017979.664852:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2246 00000100:00080000:0.0:1734017979.664854:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017979.664857:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017979.664861:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017979.664902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334377600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017979.664912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734017979.664913:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734017979.664915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017979.664917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017979.664921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2246 00000100:00080000:0.0:1734017979.664925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017979.664928:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017979.664933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017979.664936:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017979/real 1734017979] req@ffff8800a934c000 x1818247334377088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017979.664942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334377728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017979.664951:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017979.664954:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017979.664955:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334377088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017979.664956:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017979.664959:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2246 00000100:00080000:0.0:1734017979.664962:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017979.664965:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734017979.664966:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1734017979.664969:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017979.664969:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017979.664974:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334377088:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734017979.664976:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334377856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017979.664982:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334377216:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017979.665001:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017979.665007:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017979/real 1734017979] req@ffff8800a934d180 x1818247334377216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017979.665018:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334377216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017979.665026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017979.665029:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017979.665033:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334377216:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017979.665056:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334377344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017979.665070:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017979.665077:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017979/real 1734017979] req@ffff8800a934dc00 x1818247334377344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017979.665088:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334377344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017979.665095:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017979.665097:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017979.665100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334377344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017979.665106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334377472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017979.665118:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017979.665124:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017979/real 1734017979] req@ffff8800a934d880 x1818247334377472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017979.665133:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334377472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017979.665141:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017979.665143:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017979.665146:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334377472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017979.665151:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334377600:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017979.665167:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017979.665174:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017979/real 1734017979] req@ffff8800a934c700 x1818247334377600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017979.665184:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334377600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017979.665191:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017979.665193:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017979.665196:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334377600:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017979.665202:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334377728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017979.665213:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017979.665219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017979/real 1734017979] req@ffff8800a934ed80 x1818247334377728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017979.665230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334377728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017979.665237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017979.665239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017979.665242:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334377728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017979.665247:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334377856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017979.665259:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017979.665265:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017979/real 1734017979] req@ffff8800a934ca80 x1818247334377856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017979.665281:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334377856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017979.665289:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017979.665291:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017979.665294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334377856:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017980.304757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017981.306765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017982.308754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017983.310744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017984.312761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017984.672601:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017984.672611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017984.672616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017984.672627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2251 00000100:00080000:0.0:1734017984.672636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017984.672644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017984.672680:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334377984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017984.672724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017984.672728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017984.672730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017984.672734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2251 00000100:00080000:0.0:1734017984.672737:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017984.672741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017984.672751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017984.672761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334378112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017984.672770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017984.672774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017984.672775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017984.672779:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2251 00000100:00080000:0.0:1734017984.672782:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017984.672818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017984.672824:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017984.672834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334378240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017984.672843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017984.672847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017984.672848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017984.672852:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2251 00000100:00080000:0.0:1734017984.672854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017984.672859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017984.672863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017984.672870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334378368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017984.672880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017984.672883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017984.672884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017984.672888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2251 00000100:00080000:0.0:1734017984.672890:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017984.672893:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017984.672897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734017984.672898:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334377984:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734017984.672905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334378496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017984.672913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017984.672917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017984.672919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017984.672922:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2251 00000100:00080000:0.0:1734017984.672925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017984.672928:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017984.672932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017984.672941:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334378624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017984.672949:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017984.672953:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017984.672954:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017984.672958:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2251 00000100:00080000:0.0:1734017984.672975:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017984.672979:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017984.672983:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017984.672991:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334378752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017984.672994:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017984.673024:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017984/real 1734017984] req@ffff8800a934c380 x1818247334377984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017984.673046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334377984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017984.673059:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017984.673063:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017984.673069:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334377984:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017984.673076:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334378112:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017984.673096:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017984.673103:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017984/real 1734017984] req@ffff8800a934ea00 x1818247334378112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017984.673114:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334378112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017984.673122:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017984.673126:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017984.673130:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334378112:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017984.673137:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334378240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017984.673153:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017984.673159:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017984/real 1734017984] req@ffff8800a934f100 x1818247334378240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017984.673169:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334378240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017984.673177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017984.673180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017984.673184:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334378240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017984.673214:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334378368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017984.673228:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017984.673234:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017984/real 1734017984] req@ffff8800a934e300 x1818247334378368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017984.673245:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334378368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017984.673254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017984.673256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017984.673259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334378368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017984.673265:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334378496:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017984.673282:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017984.673288:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017984/real 1734017984] req@ffff8800a934e680 x1818247334378496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017984.673299:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334378496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017984.673306:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017984.673309:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017984.673312:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334378496:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017984.673318:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334378624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017984.673330:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017984.673337:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017984/real 1734017984] req@ffff8800a934ce00 x1818247334378624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017984.673347:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334378624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017984.673355:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017984.673357:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017984.673360:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334378624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017984.673366:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334378752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017984.673379:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017984.673385:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017984/real 1734017984] req@ffff8800a934df80 x1818247334378752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017984.673401:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334378752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017984.673408:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017984.673411:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017984.673414:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334378752:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017985.314760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017986.316754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017987.318785:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017988.320784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017989.322803:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017989.680598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017989.680608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017989.680612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017989.680623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2256 00000100:00080000:0.0:1734017989.680633:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017989.680641:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017989.680679:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334378880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017989.680748:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017989.680752:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017989.680755:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017989.680759:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2256 00000100:00080000:0.0:1734017989.680763:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017989.680767:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017989.680778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017989.680790:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334379008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017989.680802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017989.680805:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017989.680807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017989.680811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2256 00000100:00080000:0.0:1734017989.680813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017989.680817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017989.680822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017989.680832:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334379136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017989.680841:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017989.680844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017989.680846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017989.680849:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2256 00000100:00080000:0.0:1734017989.680852:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017989.680855:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017989.680859:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017989.680867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334379264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017989.680876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017989.680879:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017989.680881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017989.680884:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2256 00000100:00080000:0.0:1734017989.680887:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017989.680891:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017989.680893:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334378880:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734017989.680895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017989.680903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334379392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017989.680911:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017989.680914:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017989.680915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017989.680918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2256 00000100:00080000:0.0:1734017989.680922:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017989.680925:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017989.680929:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017989.680937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334379520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017989.680945:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017989.680948:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017989.680949:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017989.680953:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2256 00000100:00080000:0.0:1734017989.680967:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017989.680970:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017989.680975:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017989.680982:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334379648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017989.681015:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334379008:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017989.681038:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334379136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017989.681066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334379264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017989.681085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334379392:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017989.681102:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334379520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017989.681119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334379648:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017990.324780:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017991.326750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017992.328732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734017992.686787:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734017992.686835:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734017992.686853:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017992.686890:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017992.686898:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017992.686905:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017992.686912:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017992.686919:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734017992.686925:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734017992.687021:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017989/real 1734017992] req@ffff8800a934ce00 x1818247334379264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017992.687061:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334379264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017992.687081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017992.687087:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017992.687096:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334379264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017992.687109:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017989/real 1734017992] req@ffff8800a934e680 x1818247334379392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017992.687132:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334379392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017992.687145:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017992.687149:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017992.687156:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334379392:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017992.687168:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017989/real 1734017992] req@ffff8800a934e300 x1818247334379520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017992.687187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334379520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017992.687200:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017992.687205:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017992.687210:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334379520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017992.687222:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017989/real 1734017992] req@ffff8800a934f100 x1818247334379648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017992.687242:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334379648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017992.687254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017992.687259:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017992.687265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334379648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017992.687281:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017989/real 1734017992] req@ffff8800a934fb80 x1818247334378880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017992.687302:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334378880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017992.687327:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017992.687332:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017992.687337:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334378880:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017992.687349:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017989/real 1734017992] req@ffff8800a934d500 x1818247334379008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017992.687368:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334379008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017992.687380:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017992.687385:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017992.687390:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334379008:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017992.687400:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017989/real 1734017992] req@ffff8800a934df80 x1818247334379136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017992.687418:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334379136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017992.687430:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017992.687434:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017992.687440:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334379136:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017993.330782:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017994.332707:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017994.688608:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017994.688642:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017994.688646:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017994.688657:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2261 00000100:00080000:0.0:1734017994.688665:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017994.688673:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017994.688710:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334379776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017994.688743:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017994.688747:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017994.688749:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017994.688754:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2261 00000100:00080000:0.0:1734017994.688757:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017994.688761:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017994.688771:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017994.688782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334379904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017994.688792:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017994.688795:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017994.688797:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017994.688801:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2261 00000100:00080000:0.0:1734017994.688803:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017994.688807:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017994.688813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017994.688821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334380032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017994.688830:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017994.688833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017994.688835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017994.688839:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2261 00000100:00080000:0.0:1734017994.688841:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017994.688844:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017994.688848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017994.688855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334380160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017994.688864:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017994.688866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017994.688868:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017994.688871:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2261 00000100:00080000:0.0:1734017994.688873:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017994.688877:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017994.688881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017994.688889:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334380288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017994.688899:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017994.688901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017994.688903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017994.688907:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2261 00000100:00080000:0.0:1734017994.688910:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017994.688912:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334379776:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734017994.688914:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017994.688918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017994.688926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334380416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017994.688934:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017994.688937:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017994.688938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017994.688941:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2261 00000100:00080000:0.0:1734017994.688944:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017994.688947:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017994.688951:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017994.688958:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334380544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017994.689011:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017994.689039:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017994/real 1734017994] req@ffff8800a934ea00 x1818247334379776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017994.689061:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334379776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017994.689073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017994.689077:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017994.689082:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334379776:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017994.689090:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334379904:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017994.689109:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017994.689116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017994/real 1734017994] req@ffff8800a934c380 x1818247334379904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017994.689128:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334379904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017994.689136:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017994.689139:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017994.689142:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334379904:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017994.689149:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334380032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017994.689163:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017994.689169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017994/real 1734017994] req@ffff8800a934ca80 x1818247334380032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017994.689180:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334380032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017994.689189:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017994.689191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017994.689195:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334380032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017994.689201:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334380160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017994.689214:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017994.689220:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017994/real 1734017994] req@ffff8800a934ed80 x1818247334380160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017994.689230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334380160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017994.689237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017994.689239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017994.689243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334380160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017994.689276:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334380288:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017994.689290:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017994.689297:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017994/real 1734017994] req@ffff8800a934c700 x1818247334380288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017994.689308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334380288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017994.689315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017994.689317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017994.689320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334380288:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017994.689327:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334380416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017994.689341:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017994.689347:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017994/real 1734017994] req@ffff8800a934d880 x1818247334380416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017994.689358:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334380416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017994.689365:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017994.689367:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017994.689370:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334380416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017994.689376:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334380544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017994.689389:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017994.689395:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017994/real 1734017994] req@ffff8800a934dc00 x1818247334380544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017994.689404:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334380544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017994.689411:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017994.689414:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017994.689417:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334380544:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734017995.334748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017996.336769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017997.338766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017998.340780:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017999.342746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734017999.696642:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017999.696652:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734017999.696657:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017999.696667:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2266 00000100:00080000:0.0:1734017999.696676:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017999.696683:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734017999.696718:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334380672/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017999.696762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017999.696766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017999.696768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017999.696772:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2266 00000100:00080000:0.0:1734017999.696776:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017999.696780:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017999.696790:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017999.696801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334380800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017999.696810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017999.696813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017999.696815:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017999.696819:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2266 00000100:00080000:0.0:1734017999.696822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017999.696825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017999.696830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734017999.696839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334380928/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017999.696847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017999.696850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017999.696852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017999.696856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2266 00000100:00080000:0.0:1734017999.696859:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017999.696863:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017999.696867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734017999.696875:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334381056/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017999.696883:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017999.696886:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017999.696887:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017999.696890:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2266 00000100:00080000:0.0:1734017999.696893:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017999.696896:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017999.696900:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017999.696908:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334381184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017999.696917:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017999.696920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734017999.696921:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017999.696925:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2266 00000100:00080000:0.0:1734017999.696927:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017999.696931:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734017999.696934:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334380672:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734017999.696935:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017999.696944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334381312/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017999.696951:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017999.696955:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017999.696957:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017999.696960:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2266 00000100:00080000:0.0:1734017999.696963:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017999.696966:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734017999.696970:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017999.696978:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334381440/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017999.697035:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017999.697065:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017999/real 1734017999] req@ffff8800a934d180 x1818247334380672/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017999.697088:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334380672/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017999.697101:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017999.697105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017999.697110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334380672:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734017999.697119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334380800:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017999.697138:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017999.697145:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017999/real 1734017999] req@ffff8800a934c000 x1818247334380800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017999.697157:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334380800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017999.697165:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017999.697168:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017999.697171:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334380800:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017999.697178:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334380928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017999.697193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017999.697199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017999/real 1734017999] req@ffff8800a934dc00 x1818247334380928/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017999.697209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334380928/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017999.697217:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017999.697220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017999.697223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334380928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017999.697230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334381056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017999.697244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017999.697250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017999/real 1734017999] req@ffff8800a934d880 x1818247334381056/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017999.697260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334381056/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017999.697268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017999.697270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017999.697273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334381056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017999.697305:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334381184:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734017999.697319:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017999.697325:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017999/real 1734017999] req@ffff8800a934c700 x1818247334381184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017999.697335:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334381184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017999.697343:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017999.697345:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017999.697348:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334381184:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734017999.697355:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334381312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017999.697371:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017999.697378:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017999/real 1734017999] req@ffff8800a934ed80 x1818247334381312/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017999.697389:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334381312/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017999.697396:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017999.697398:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017999.697401:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334381312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734017999.697407:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334381440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734017999.697420:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734017999.697426:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017999/real 1734017999] req@ffff8800a934ca80 x1818247334381440/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017999.697436:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334381440/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017999.697443:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017999.697445:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734017999.697448:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334381440:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018000.344757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018001.346767:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018002.348724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018003.350758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018004.352776:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018004.704601:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018004.704611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018004.704615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018004.704626:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2271 00000100:00080000:0.0:1734018004.704635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018004.704643:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018004.704681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334381568/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018004.704717:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018004.704721:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018004.704723:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018004.704728:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2271 00000100:00080000:0.0:1734018004.704731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018004.704735:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018004.704745:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018004.704757:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334381696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018004.704768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018004.704771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018004.704772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018004.704776:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2271 00000100:00080000:0.0:1734018004.704779:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018004.704782:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018004.704787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018004.704795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334381824/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018004.704804:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018004.704808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018004.704809:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018004.704813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2271 00000100:00080000:0.0:1734018004.704815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018004.704818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018004.704822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018004.704830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334381952/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018004.704839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018004.704842:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018004.704843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018004.704846:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2271 00000100:00080000:0.0:1734018004.704849:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018004.704852:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018004.704857:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018004.704866:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334382080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018004.704871:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334381568:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018004.704875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018004.704878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018004.704879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018004.704883:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2271 00000100:00080000:0.0:1734018004.704885:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018004.704888:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018004.704892:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018004.704902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334382208/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018004.704909:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018004.704912:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018004.704914:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018004.704917:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2271 00000100:00080000:0.0:1734018004.704920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018004.704923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018004.704927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018004.704934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334382336/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018004.704972:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018004.705002:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018004/real 1734018004] req@ffff8800a934c380 x1818247334381568/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018004.705033:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334381568/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018004.705045:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018004.705049:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018004.705054:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334381568:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018004.705062:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334381696:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018004.705081:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018004.705088:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018004/real 1734018004] req@ffff8800a934ea00 x1818247334381696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018004.705099:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334381696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018004.705108:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018004.705110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018004.705114:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334381696:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018004.705120:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334381824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018004.705136:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018004.705142:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018004/real 1734018004] req@ffff8800a934df80 x1818247334381824/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018004.705152:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334381824/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018004.705160:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018004.705163:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018004.705167:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334381824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018004.705174:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334381952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018004.705187:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018004.705193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018004/real 1734018004] req@ffff8800a934d500 x1818247334381952/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018004.705203:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334381952/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018004.705211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018004.705213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018004.705216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334381952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018004.705248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334382080:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018004.705262:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018004.705268:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018004/real 1734018004] req@ffff8800a934fb80 x1818247334382080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018004.705278:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334382080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018004.705286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018004.705288:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018004.705291:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334382080:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018004.705297:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334382208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018004.705314:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018004.705320:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018004/real 1734018004] req@ffff8800a934f100 x1818247334382208/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018004.705331:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334382208/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018004.705338:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018004.705340:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018004.705343:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334382208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018004.705349:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334382336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018004.705362:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018004.705368:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018004/real 1734018004] req@ffff8800a934e300 x1818247334382336/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018004.705378:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334382336/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018004.705385:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018004.705387:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018004.705390:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334382336:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018005.354726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018006.356705:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018007.358776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018008.360741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018009.362735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018009.712629:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018009.712639:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018009.712643:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018009.712654:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2276 00000100:00080000:0.0:1734018009.712663:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018009.712671:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018009.712706:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334382464/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018009.712739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018009.712743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018009.712745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018009.712750:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2276 00000100:00080000:0.0:1734018009.712753:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018009.712757:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018009.712767:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018009.712778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334382592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018009.712788:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018009.712791:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018009.712793:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018009.712797:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2276 00000100:00080000:0.0:1734018009.712800:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018009.712803:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018009.712809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018009.712820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334382720/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018009.712829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018009.712832:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018009.712833:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018009.712837:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2276 00000100:00080000:0.0:1734018009.712839:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018009.712840:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334382464:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018009.712843:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018009.712847:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018009.712854:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334382848/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018009.712863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018009.712866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018009.712868:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018009.712871:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2276 00000100:00080000:0.0:1734018009.712873:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018009.712876:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018009.712880:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018009.712888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334382976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018009.712896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018009.712898:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018009.712900:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018009.712904:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2276 00000100:00080000:0.0:1734018009.712907:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018009.712908:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018009.712911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018009.712915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018009.712923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334383104/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018009.712930:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018009/real 1734018009] req@ffff8800a934e680 x1818247334382464/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018009.712931:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018009.712934:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018009.712935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018009.712939:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2276 00000100:00080000:0.0:1734018009.712941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018009.712944:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018009.712948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018009.712956:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334383232/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018009.712958:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334382464/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018009.712967:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018009.712971:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018009.712975:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334382464:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018009.712982:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334382592:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018009.712999:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018009.713006:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018009/real 1734018009] req@ffff8800a934ce00 x1818247334382592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018009.713017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334382592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018009.713025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018009.713027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018009.713030:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334382592:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018009.713053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334382720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018009.713067:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018009.713074:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018009/real 1734018009] req@ffff8800a934e300 x1818247334382720/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018009.713084:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334382720/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018009.713092:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018009.713094:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018009.713098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334382720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018009.713104:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334382848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018009.713117:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018009.713123:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018009/real 1734018009] req@ffff8800a934f100 x1818247334382848/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018009.713132:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334382848/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018009.713140:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018009.713142:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018009.713145:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334382848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018009.713150:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334382976:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018009.713162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018009.713168:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018009/real 1734018009] req@ffff8800a934fb80 x1818247334382976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018009.713178:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334382976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018009.713185:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018009.713187:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018009.713190:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334382976:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018009.713196:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334383104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018009.713212:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018009.713218:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018009/real 1734018009] req@ffff8800a934d500 x1818247334383104/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018009.713229:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334383104/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018009.713236:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018009.713238:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018009.713241:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334383104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018009.713247:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334383232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018009.713259:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018009.713265:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018009/real 1734018009] req@ffff8800a934df80 x1818247334383232/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018009.713274:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334383232/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018009.713281:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018009.713283:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018009.713286:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334383232:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018010.364742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018011.366760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018012.368735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018013.370751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018014.372717:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018014.720634:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018014.720647:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018014.720652:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018014.720667:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2281 00000100:00080000:0.0:1734018014.720678:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018014.720688:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018014.720736:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334383360/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018014.720783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018014.720788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018014.720790:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018014.720795:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2281 00000100:00080000:0.0:1734018014.720798:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018014.720802:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018014.720813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018014.720825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334383488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018014.720835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018014.720838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018014.720840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018014.720844:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2281 00000100:00080000:0.0:1734018014.720848:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018014.720852:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018014.720857:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018014.720866:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334383616/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018014.720875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018014.720878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018014.720880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018014.720884:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2281 00000100:00080000:0.0:1734018014.720886:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018014.720890:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018014.720895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018014.720903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334383744/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018014.720913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018014.720914:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334383360:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018014.720916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018014.720918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018014.720922:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2281 00000100:00080000:0.0:1734018014.720924:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018014.720928:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018014.720932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018014.720943:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334383872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018014.720951:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018014.720955:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018014.720956:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018014.720960:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2281 00000100:00080000:0.0:1734018014.720973:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018014.720977:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018014.720982:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018014.720988:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018014.720991:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334384000/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018014.721000:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018014.721004:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018014.721006:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018014.721010:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2281 00000100:00100000:1.0:1734018014.721012:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018014/real 1734018014] req@ffff8800a934ea00 x1818247334383360/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018014.721014:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018014.721017:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018014.721022:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018014.721029:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334384128/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018014.721040:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334383360/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018014.721052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018014.721055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018014.721060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334383360:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018014.721069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334383488:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018014.721087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018014.721094:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018014/real 1734018014] req@ffff8800a934c380 x1818247334383488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018014.721105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334383488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018014.721113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018014.721115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018014.721119:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334383488:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018014.721125:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334383616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018014.721139:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018014.721145:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018014/real 1734018014] req@ffff8800a934ca80 x1818247334383616/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018014.721156:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334383616/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018014.721163:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018014.721166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018014.721170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334383616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018014.721196:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334383744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018014.721210:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018014.721216:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018014/real 1734018014] req@ffff8800a934ed80 x1818247334383744/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018014.721227:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334383744/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018014.721234:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018014.721237:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018014.721240:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334383744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018014.721246:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334383872:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018014.721258:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018014.721264:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018014/real 1734018014] req@ffff8800a934c700 x1818247334383872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018014.721319:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334383872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018014.721327:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018014.721330:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018014.721333:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334383872:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018014.721340:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334384000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018014.721360:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018014.721368:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018014/real 1734018014] req@ffff8800a934d880 x1818247334384000/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018014.721378:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334384000/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018014.721386:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018014.721389:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018014.721392:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334384000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018014.721398:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334384128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018014.721411:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018014.721417:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018014/real 1734018014] req@ffff8800a934dc00 x1818247334384128/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018014.721427:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334384128/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018014.721434:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018014.721437:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018014.721440:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334384128:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018015.374748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018016.376735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018017.378779:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018018.380719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018019.382731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018019.728614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018019.728626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018019.728631:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018019.728645:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2286 00000100:00080000:0.0:1734018019.728655:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018019.728664:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018019.728708:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334384256/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018019.728746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018019.728751:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018019.728753:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018019.728758:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2286 00000100:00080000:0.0:1734018019.728761:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018019.728765:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018019.728776:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018019.728787:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334384384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018019.728799:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018019.728802:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018019.728804:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018019.728808:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2286 00000100:00080000:0.0:1734018019.728810:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018019.728814:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018019.728818:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018019.728830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334384512/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018019.728839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018019.728842:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018019.728844:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018019.728847:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2286 00000100:00080000:0.0:1734018019.728850:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018019.728853:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018019.728857:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018019.728866:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334384640/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018019.728876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018019.728879:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018019.728881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018019.728884:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2286 00000100:00100000:1.0:1734018019.728884:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334384256:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018019.728888:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018019.728891:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018019.728896:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018019.728905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334384768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018019.728913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018019.728916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018019.728918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018019.728921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2286 00000100:00080000:0.0:1734018019.728924:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018019.728927:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018019.728931:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018019.728940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334384896/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018019.728947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018019.728951:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018019.728952:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018019.728956:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2286 00000100:00080000:0.0:1734018019.728959:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018019.728959:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018019.728963:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018019.728967:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018019.728974:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334385024/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018019.728984:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018019/real 1734018019] req@ffff8800a934c000 x1818247334384256/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018019.729012:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334384256/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018019.729024:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018019.729027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018019.729032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334384256:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018019.729040:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334384384:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018019.729057:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018019.729064:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018019/real 1734018019] req@ffff8800a934d180 x1818247334384384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018019.729075:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334384384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018019.729083:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018019.729086:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018019.729089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334384384:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018019.729096:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334384512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018019.729109:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018019.729116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018019/real 1734018019] req@ffff8800a934dc00 x1818247334384512/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018019.729126:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334384512/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018019.729133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018019.729136:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018019.729140:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334384512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018019.729166:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334384640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018019.729180:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018019.729232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018019/real 1734018019] req@ffff8800a934d880 x1818247334384640/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018019.729245:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334384640/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018019.729253:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018019.729256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018019.729260:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334384640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018019.729267:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334384768:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018019.729283:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018019.729290:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018019/real 1734018019] req@ffff8800a934c700 x1818247334384768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018019.729301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334384768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018019.729309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018019.729311:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018019.729314:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334384768:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018019.729320:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334384896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018019.729336:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018019.729343:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018019/real 1734018019] req@ffff8800a934ed80 x1818247334384896/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018019.729354:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334384896/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018019.729361:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018019.729364:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018019.729367:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334384896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018019.729372:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334385024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018019.729385:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018019.729391:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018019/real 1734018019] req@ffff8800a934ca80 x1818247334385024/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018019.729401:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334385024/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018019.729408:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018019.729411:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018019.729414:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334385024:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018020.384746:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018021.386752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018022.388808:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018023.390628:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018024.392662:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018024.736575:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018024.736582:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018024.736585:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018024.736593:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2291 00000100:00080000:0.0:1734018024.736599:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018024.736605:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018024.736633:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334385152/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018024.736655:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018024.736657:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018024.736658:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018024.736660:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2291 00000100:00080000:0.0:1734018024.736662:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018024.736664:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018024.736671:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018024.736676:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334385280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018024.736681:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018024.736683:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018024.736684:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018024.736686:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2291 00000100:00080000:0.0:1734018024.736688:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018024.736690:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018024.736693:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018024.736698:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334385408/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018024.736702:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018024.736705:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018024.736705:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018024.736707:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2291 00000100:00080000:0.0:1734018024.736708:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018024.736710:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018024.736712:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018024.736716:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334385536/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018024.736720:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018024.736722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018024.736723:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018024.736724:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2291 00000100:00080000:0.0:1734018024.736726:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018024.736727:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018024.736730:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018024.736733:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334385664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018024.736739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018024.736740:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018024.736741:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018024.736742:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334385152:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018024.736743:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2291 00000100:00080000:0.0:1734018024.736745:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018024.736747:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018024.736749:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018024.736755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334385792/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018024.736759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018024.736760:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018024.736761:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018024.736763:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2291 00000100:00080000:0.0:1734018024.736764:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018024.736766:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018024.736768:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018024.736772:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334385920/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018024.736806:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018024.736826:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018024/real 1734018024] req@ffff8800a934c380 x1818247334385152/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018024.736843:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334385152/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018024.736850:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018024.736852:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018024.736855:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334385152:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018024.736860:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334385280:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018024.736870:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018024.736873:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018024/real 1734018024] req@ffff8800a934ea00 x1818247334385280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018024.736879:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334385280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018024.736884:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018024.736885:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018024.736887:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334385280:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018024.736890:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334385408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018024.736899:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018024.736902:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018024/real 1734018024] req@ffff8800a934df80 x1818247334385408/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018024.736907:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334385408/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018024.736911:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018024.736913:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018024.736915:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334385408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018024.736919:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334385536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018024.736927:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018024.736930:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018024/real 1734018024] req@ffff8800a934d500 x1818247334385536/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018024.736935:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334385536/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018024.736939:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018024.736940:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018024.736941:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334385536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018024.736961:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334385664:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018024.736969:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018024.736972:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018024/real 1734018024] req@ffff8800a934fb80 x1818247334385664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018024.736978:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334385664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018024.736982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018024.736984:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018024.736985:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334385664:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018024.736988:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334385792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018024.736997:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018024.737000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018024/real 1734018024] req@ffff8800a934f100 x1818247334385792/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018024.737006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334385792/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018024.737010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018024.737011:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018024.737012:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334385792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018024.737015:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334385920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018024.737022:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018024.737026:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018024/real 1734018024] req@ffff8800a934e300 x1818247334385920/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018024.737031:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334385920/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018024.737034:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018024.737035:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018024.737037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334385920:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018025.394749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018026.396684:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018027.398682:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018028.400664:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018029.402681:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018029.744578:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018029.744588:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018029.744592:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018029.744600:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2296 00000100:00080000:0.0:1734018029.744608:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018029.744614:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018029.744643:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334386048/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018029.744676:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018029.744679:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018029.744680:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018029.744684:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2296 00000100:00080000:0.0:1734018029.744686:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018029.744690:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018029.744698:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018029.744709:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334386176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018029.744717:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018029.744719:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018029.744720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018029.744723:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2296 00000100:00080000:0.0:1734018029.744726:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018029.744729:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018029.744732:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018029.744741:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334386304/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018029.744746:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334386048:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018029.744749:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018029.744751:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018029.744753:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018029.744755:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2296 00000100:00080000:0.0:1734018029.744758:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018029.744761:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018029.744765:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018029.744771:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334386432/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018029.744778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018029.744780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018029.744782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018029.744784:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2296 00000100:00080000:0.0:1734018029.744787:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018029.744789:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018029.744792:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018029.744793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018029.744799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334386560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018029.744805:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018029/real 1734018029] req@ffff8800a934ce00 x1818247334386048/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018029.744806:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018029.744809:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018029.744810:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018029.744813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2296 00000100:00080000:0.0:1734018029.744815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018029.744817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018029.744820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018029.744820:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334386048/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018029.744826:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334386688/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018029.744826:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018029.744828:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018029.744830:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334386048:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018029.744832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018029.744834:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334386176:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734018029.744835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018029.744837:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018029.744839:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2296 00000100:00080000:0.0:1734018029.744843:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018029.744844:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018029.744845:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018029.744847:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018029/real 1734018029] req@ffff8800a934e680 x1818247334386176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018029.744848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018029.744853:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334386176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018029.744856:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334386816/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018029.744857:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018029.744858:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018029.744860:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334386176:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018029.744874:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334386304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018029.744881:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018029.744884:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018029/real 1734018029] req@ffff8800a934e300 x1818247334386304/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018029.744889:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334386304/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018029.744893:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018029.744895:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018029.744896:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334386304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018029.744900:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334386432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018029.744906:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018029.744909:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018029/real 1734018029] req@ffff8800a934f100 x1818247334386432/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018029.744914:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334386432/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018029.744918:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018029.744920:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018029.744921:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334386432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018029.744924:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334386560:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018029.744930:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018029.744933:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018029/real 1734018029] req@ffff8800a934fb80 x1818247334386560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018029.744938:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334386560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018029.744942:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018029.744943:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018029.744945:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334386560:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018029.744948:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334386688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018029.744956:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018029.744959:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018029/real 1734018029] req@ffff8800a934d500 x1818247334386688/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018029.744964:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334386688/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018029.744968:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018029.744969:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018029.744970:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334386688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018029.744973:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334386816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018029.744979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018029.744982:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018029/real 1734018029] req@ffff8800a934df80 x1818247334386816/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018029.744987:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334386816/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018029.744991:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018029.744992:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018029.744993:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334386816:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018030.404653:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018031.406649:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018032.408662:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018033.410679:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018034.412673:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018034.752590:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018034.752599:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018034.752604:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018034.752613:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2301 00000100:00080000:0.0:1734018034.752623:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018034.752630:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018034.752661:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334386944/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018034.752709:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018034.752712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018034.752717:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018034.752721:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2301 00000100:00080000:0.0:1734018034.752724:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018034.752727:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018034.752736:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018034.752746:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334387072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018034.752750:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334386944:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018034.752756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018034.752759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018034.752760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018034.752763:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2301 00000100:00080000:0.0:1734018034.752767:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018034.752770:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018034.752775:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018034.752783:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334387200/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018034.752791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018034.752794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734018034.752794:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018034.752795:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018034.752799:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2301 00000100:00080000:0.0:1734018034.752802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018034.752808:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018034/real 1734018034] req@ffff8800a934ea00 x1818247334386944/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018034.752823:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334386944/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018034.752829:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018034.752830:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018034.752833:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334386944:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018034.752844:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018034.752846:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334387072:192.168.201.116@tcp:38:kworker.0 00000008:00100000:0.0:1734018034.752848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000800:00000100:1.0:1734018034.752856:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018034.752857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334387328/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018034.752860:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018034/real 1734018034] req@ffff8800a934c380 x1818247334387072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018034.752865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018034.752866:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334387072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018034.752868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018034.752870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018034.752870:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018034.752871:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734018034.752873:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2301 00000100:00100000:1.0:1734018034.752873:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334387072:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018034.752876:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334387200:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734018034.752877:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018034.752881:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018034.752883:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018034.752884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018034.752887:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018034/real 1734018034] req@ffff8800a934ca80 x1818247334387200/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018034.752892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334387456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018034.752892:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334387200/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018034.752896:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018034.752897:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734018034.752899:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018034.752899:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334387200:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734018034.752902:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018034.752903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018034.752906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2301 00000100:00100000:1.0:1734018034.752906:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334387328:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734018034.752908:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018034.752911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018034.752913:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018034.752915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018034.752916:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018034/real 1734018034] req@ffff8800a934ed80 x1818247334387328/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018034.752922:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334387328/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018034.752924:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334387584/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018034.752926:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018034.752927:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018034.752929:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334387328:192.168.201.116@tcp:8:kworker.0 00000100:00080000:0.0:1734018034.752945:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018034.752948:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018034.752949:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018034.752952:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2301 00000100:00080000:0.0:1734018034.752955:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018034.752958:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018034.752962:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018034.752966:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334387456:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018034.752969:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334387712/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018034.752982:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018034.752988:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018034/real 1734018034] req@ffff8800a934c700 x1818247334387456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018034.752997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334387456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018034.753003:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018034.753005:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018034.753008:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334387456:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018034.753020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334387584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018034.753037:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018034.753043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018034/real 1734018034] req@ffff8800a934d880 x1818247334387584/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018034.753052:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334387584/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018034.753059:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018034.753060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018034.753063:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334387584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018034.753067:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334387712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018034.753087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018034.753093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018034/real 1734018034] req@ffff8800a934dc00 x1818247334387712/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018034.753103:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334387712/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018034.753109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018034.753110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018034.753113:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334387712:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018035.414733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018036.416620:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018037.418678:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018038.420641:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018039.422641:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018039.760550:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018039.760559:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018039.760562:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018039.760569:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2306 00000100:00080000:0.0:1734018039.760575:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018039.760580:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018039.760607:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334387840/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018039.760629:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018039.760632:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018039.760633:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018039.760636:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2306 00000100:00080000:0.0:1734018039.760639:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018039.760642:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018039.760650:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018039.760660:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334387968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018039.760668:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018039.760669:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334387840:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018039.760670:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018039.760671:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018039.760674:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2306 00000100:00080000:0.0:1734018039.760676:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018039.760680:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018039.760683:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018039.760692:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334388096/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018039.760700:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018039.760702:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018039.760703:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018039.760706:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2306 00000100:00080000:0.0:1734018039.760708:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018039.760710:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018039.760714:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000800:00000100:1.0:1734018039.760715:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018039.760721:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334388224/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018039.760727:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018039.760730:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018039.760731:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018039.760731:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018039/real 1734018039] req@ffff8800a934d180 x1818247334387840/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018039.760734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2306 00000100:00080000:0.0:1734018039.760736:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018039.760739:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018039.760743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018039.760749:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334388352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018039.760751:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334387840/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018039.760756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018039.760758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1734018039.760759:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018039.760760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018039.760761:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734018039.760763:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2306 00000100:00100000:1.0:1734018039.760764:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334387840:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018039.760765:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018039.760768:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018039.760771:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018039.760777:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334388480/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018039.760781:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334387968:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734018039.760784:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018039.760787:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018039.760788:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018039.760790:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2306 00000800:00000100:1.0:1734018039.760796:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018039.760799:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018039.760802:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018039/real 1734018039] req@ffff8800a934c000 x1818247334387968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018039.760803:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018039.760806:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018039.760811:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334387968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018039.760813:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334388608/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018039.760818:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018039.760820:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018039.760823:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334387968:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018039.760828:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334388096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018039.760841:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018039.760875:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018039/real 1734018039] req@ffff8800a934dc00 x1818247334388096/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018039.760885:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334388096/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018039.760892:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018039.760894:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018039.760897:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334388096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018039.760901:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334388224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018039.760914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018039.760918:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018039/real 1734018039] req@ffff8800a934d880 x1818247334388224/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018039.760924:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334388224/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018039.760928:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018039.760929:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018039.760930:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334388224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018039.760933:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334388352:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018039.760940:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018039.760943:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018039/real 1734018039] req@ffff8800a934c700 x1818247334388352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018039.760948:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334388352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018039.760952:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018039.760953:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018039.760955:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334388352:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018039.760964:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334388480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018039.760973:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018039.760977:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018039/real 1734018039] req@ffff8800a934ed80 x1818247334388480/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018039.760982:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334388480/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018039.760986:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018039.760987:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018039.760989:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334388480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018039.760991:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334388608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018039.760998:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018039.761001:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018039/real 1734018039] req@ffff8800a934ca80 x1818247334388608/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018039.761006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334388608/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018039.761009:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018039.761010:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018039.761011:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334388608:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018040.424575:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018041.426608:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018042.428672:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018043.430672:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018044.432683:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018044.768638:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018044.768650:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018044.768655:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018044.768666:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2311 00000100:00080000:0.0:1734018044.768676:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018044.768684:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018044.768723:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334388736/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018044.768798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018044.768803:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018044.768805:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018044.768809:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2311 00000100:00080000:0.0:1734018044.768813:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018044.768817:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018044.768829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018044.768843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334388864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018044.768853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018044.768857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018044.768858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018044.768862:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2311 00000100:00080000:0.0:1734018044.768865:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018044.768869:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018044.768874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018044.768882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334388992/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018044.768892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018044.768895:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018044.768896:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018044.768900:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2311 00000100:00080000:0.0:1734018044.768902:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018044.768902:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334388736:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018044.768906:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018044.768910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018044.768918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334389120/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018044.768926:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018044.768929:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018044.768931:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018044.768934:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2311 00000100:00080000:0.0:1734018044.768936:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018044.768939:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018044.768943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018044.768950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334389248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018044.768958:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018044.768961:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018044.768962:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018044.768985:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018044.769011:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018044/real 1734018044] req@ffff8800a934c380 x1818247334388736/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018044.769039:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334388736/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018044.769049:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018044.769052:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018044.769057:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334388736:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018044.769065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334388864:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734018044.769077:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2311 00000100:00080000:0.0:1734018044.769083:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018044.769085:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018044.769087:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018044.769092:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018044.769093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018044/real 1734018044] req@ffff8800a934ea00 x1818247334388864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018044.769103:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334389376/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018044.769104:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334388864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018044.769112:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734018044.769112:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018044.769114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734018044.769116:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018044.769117:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018044.769118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334388864:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734018044.769121:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2311 00000100:00080000:0.0:1734018044.769124:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018044.769127:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018044.769131:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018044.769138:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334389504/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018044.769153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334388992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018044.769169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018044.769176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018044/real 1734018044] req@ffff8800a934df80 x1818247334388992/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018044.769186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334388992/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018044.769194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018044.769197:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018044.769201:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334388992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018044.769207:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334389120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018044.769219:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018044.769225:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018044/real 1734018044] req@ffff8800a934d500 x1818247334389120/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018044.769236:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334389120/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018044.769243:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018044.769246:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018044.769249:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334389120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018044.769254:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334389248:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018044.769267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018044.769273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018044/real 1734018044] req@ffff8800a934fb80 x1818247334389248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018044.769283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334389248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018044.769290:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018044.769292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018044.769296:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334389248:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018044.769301:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334389376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018044.769317:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018044.769323:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018044/real 1734018044] req@ffff8800a934f100 x1818247334389376/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018044.769334:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334389376/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018044.769341:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018044.769351:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018044.769354:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334389376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018044.769360:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334389504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018044.769372:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018044.769380:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018044/real 1734018044] req@ffff8800a934e300 x1818247334389504/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018044.769396:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334389504/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018044.769418:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018044.769421:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018044.769426:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334389504:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018045.434797:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018046.436680:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018047.438768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018048.440663:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018049.442723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018049.776633:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018049.776643:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018049.776647:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018049.776655:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2316 00000100:00080000:0.0:1734018049.776662:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018049.776668:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018049.776756:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334389632/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018049.776786:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018049.776789:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018049.776791:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018049.776794:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2316 00000100:00080000:0.0:1734018049.776797:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018049.776800:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018049.776808:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018049.776817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334389760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018049.776826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018049.776828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018049.776829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018049.776832:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2316 00000100:00080000:0.0:1734018049.776834:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018049.776837:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018049.776841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018049.776849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334389888/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018049.776858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018049.776860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018049.776861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018049.776864:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2316 00000100:00080000:0.0:1734018049.776866:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018049.776868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018049.776872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018049.776878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334390016/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018049.776878:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334389632:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018049.776885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018049.776887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018049.776888:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018049.776891:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2316 00000100:00080000:0.0:1734018049.776893:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018049.776896:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018049.776899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018049.776905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334390144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018049.776911:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018049.776913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018049.776914:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018049.776916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2316 00000100:00080000:0.0:1734018049.776918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018049.776921:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018049.776924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018049.776930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334390272/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018049.776936:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018049.776938:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018049.776939:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018049.776941:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2316 00000100:00080000:0.0:1734018049.776944:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018049.776946:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018049.776949:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018049.776955:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334390400/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018049.776958:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018049.776982:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018049/real 1734018049] req@ffff8800a934e680 x1818247334389632/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018049.777006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334389632/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018049.777017:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018049.777021:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018049.777025:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334389632:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018049.777033:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334389760:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018049.777055:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018049.777063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018049/real 1734018049] req@ffff8800a934ce00 x1818247334389760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018049.777075:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334389760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018049.777083:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018049.777086:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018049.777088:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334389760:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018049.777117:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334389888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018049.777135:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018049.777142:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018049/real 1734018049] req@ffff8800a934e300 x1818247334389888/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018049.777151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334389888/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018049.777158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018049.777161:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018049.777164:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334389888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018049.777169:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334390016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018049.777182:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018049.777187:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018049/real 1734018049] req@ffff8800a934f100 x1818247334390016/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018049.777196:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334390016/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018049.777203:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018049.777205:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018049.777209:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334390016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018049.777215:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334390144:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018049.777232:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018049.777239:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018049/real 1734018049] req@ffff8800a934fb80 x1818247334390144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018049.777248:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334390144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018049.777255:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018049.777257:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018049.777260:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334390144:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018049.777266:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334390272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018049.777285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018049.777291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018049/real 1734018049] req@ffff8800a934d500 x1818247334390272/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018049.777301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334390272/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018049.777307:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018049.777309:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018049.777312:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334390272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018049.777318:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334390400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018049.777332:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018049.777338:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018049/real 1734018049] req@ffff8800a934df80 x1818247334390400/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018049.777347:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334390400/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018049.777354:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018049.777356:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018049.777359:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334390400:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018050.444730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018051.446718:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018052.448743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018053.450717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018054.452654:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018054.784574:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018054.784581:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018054.784584:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018054.784620:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2321 00000100:00080000:0.0:1734018054.784627:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018054.784632:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018054.784658:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334390528/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018054.784680:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018054.784682:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018054.784683:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018054.784686:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2321 00000100:00080000:0.0:1734018054.784688:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018054.784690:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018054.784696:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018054.784702:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334390656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018054.784707:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018054.784709:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018054.784710:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018054.784712:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2321 00000100:00080000:0.0:1734018054.784714:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018054.784716:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018054.784719:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018054.784724:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334390784/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018054.784728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018054.784730:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018054.784731:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018054.784732:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2321 00000100:00080000:0.0:1734018054.784734:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018054.784735:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018054.784737:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018054.784741:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334390912/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018054.784746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018054.784747:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018054.784748:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018054.784750:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2321 00000100:00080000:0.0:1734018054.784751:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018054.784753:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018054.784755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018054.784759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334391040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018054.784763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018054.784764:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018054.784765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018054.784768:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2321 00000100:00080000:0.0:1734018054.784769:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018054.784771:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018054.784773:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018054.784778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334391168/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018054.784783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018054.784784:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018054.784785:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018054.784787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2321 00000100:00080000:0.0:1734018054.784788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018054.784789:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334390528:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018054.784790:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018054.784792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018054.784796:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334391296/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018054.784912:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334390656:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018054.784932:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334390784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018054.784946:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334390912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018054.784960:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334391040:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018054.784984:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334391168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018054.784997:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334391296:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018055.454629:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018056.456716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018057.458731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018057.790828:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734018057.790882:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018057.790905:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018057.790941:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018057.790948:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018057.790954:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018057.790960:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018057.790966:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018057.790972:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734018057.791039:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018054/real 1734018057] req@ffff8800a934d880 x1818247334391168/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018057.791066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334391168/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018057.791093:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018057.791097:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018057.791102:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334391168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018057.791108:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018054/real 1734018057] req@ffff8800a934dc00 x1818247334391296/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018057.791117:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334391296/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018057.791122:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018057.791124:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018057.791127:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334391296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018057.791132:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018054/real 1734018057] req@ffff8800a934ea00 x1818247334390528/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018057.791141:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334390528/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018057.791147:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018057.791149:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018057.791152:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334390528:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018057.791157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018054/real 1734018057] req@ffff8800a934c380 x1818247334390656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018057.791168:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334390656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018057.791177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018057.791180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018057.791184:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334390656:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018057.791191:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018054/real 1734018057] req@ffff8800a934ca80 x1818247334390784/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018057.791203:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334390784/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018057.791222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018057.791225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018057.791229:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334390784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018057.791237:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018054/real 1734018057] req@ffff8800a934ed80 x1818247334390912/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018057.791250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334390912/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018057.791267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018057.791270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018057.791274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334390912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018057.791282:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018054/real 1734018057] req@ffff8800a934c700 x1818247334391040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018057.791295:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334391040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018057.791304:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018057.791307:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018057.791311:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334391040:192.168.201.116@tcp:38:kworker.0 00010000:00010000:3.0:1734018058.460840:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018059.462699:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018059.792548:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018059.792555:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018059.792558:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018059.792566:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2326 00000100:00080000:0.0:1734018059.792572:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018059.792577:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018059.792602:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334391424/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018059.792630:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018059.792632:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018059.792633:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018059.792635:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2326 00000100:00080000:0.0:1734018059.792637:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018059.792639:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018059.792645:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018059.792652:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334391552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018059.792657:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018059.792658:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018059.792659:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018059.792661:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2326 00000100:00080000:0.0:1734018059.792662:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018059.792665:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018059.792667:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018059.792673:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334391680/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018059.792674:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334391424:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018059.792678:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018059.792679:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018059.792680:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018059.792682:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2326 00000100:00080000:0.0:1734018059.792683:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018059.792686:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018059.792688:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018059.792692:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334391808/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018059.792697:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018059.792699:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018059.792699:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018059.792701:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2326 00000100:00080000:0.0:1734018059.792708:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018059.792710:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018059.792712:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018059.792716:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334391936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018059.792717:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018059.792720:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018059.792722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018059.792722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018059.792724:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2326 00000100:00080000:0.0:1734018059.792726:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018059.792727:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018059.792729:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018059/real 1734018059] req@ffff8800a934c000 x1818247334391424/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018059.792730:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018059.792734:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334392064/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018059.792738:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018059.792740:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018059.792740:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334391424/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018059.792741:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018059.792742:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2326 00000100:00080000:0.0:1734018059.792744:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018059.792746:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734018059.792746:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1734018059.792748:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018059.792748:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018059.792751:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334391424:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018059.792752:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334392192/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018059.792755:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334391552:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018059.792765:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018059.792770:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018059/real 1734018059] req@ffff8800a934d180 x1818247334391552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.792776:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334391552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.792780:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.792781:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018059.792783:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334391552:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018059.792797:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334391680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018059.792804:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018059.792807:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018059/real 1734018059] req@ffff8800a934c700 x1818247334391680/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.792812:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334391680/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.792816:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.792818:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018059.792819:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334391680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018059.792822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334391808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018059.792829:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018059.792832:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018059/real 1734018059] req@ffff8800a934ed80 x1818247334391808/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.792837:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334391808/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.792845:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.792846:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018059.792848:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334391808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018059.792851:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334391936:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018059.792857:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018059.792860:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018059/real 1734018059] req@ffff8800a934ca80 x1818247334391936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.792865:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334391936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.792869:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.792870:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018059.792872:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334391936:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018059.792875:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334392064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018059.792882:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018059.792885:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018059/real 1734018059] req@ffff8800a934c380 x1818247334392064/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.792889:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334392064/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.792893:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.792895:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018059.792897:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334392064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018059.792900:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334392192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018059.792907:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018059.792910:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018059/real 1734018059] req@ffff8800a934ea00 x1818247334392192/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.792915:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334392192/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.792918:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.792920:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018059.792921:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334392192:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018060.464709:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018061.466737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018062.468842:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018063.470715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018064.472628:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018064.800573:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018064.800581:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018064.800584:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018064.800592:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2331 00000100:00080000:0.0:1734018064.800598:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018064.800604:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018064.800632:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334392320/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018064.800654:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018064.800656:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018064.800657:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018064.800660:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2331 00000100:00080000:0.0:1734018064.800662:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018064.800664:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018064.800671:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018064.800676:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334392448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018064.800682:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018064.800684:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018064.800685:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018064.800687:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2331 00000100:00080000:0.0:1734018064.800689:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018064.800691:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018064.800692:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334392320:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018064.800693:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018064.800698:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334392576/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018064.800703:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018064.800704:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018064.800705:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018064.800707:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2331 00000100:00080000:0.0:1734018064.800708:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018064.800710:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018064.800712:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018064.800717:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334392704/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018064.800721:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018064.800723:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018064.800724:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018064.800725:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2331 00000100:00080000:0.0:1734018064.800728:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018064.800730:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018064.800732:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018064.800738:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334392832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018064.800738:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018064.800742:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018064.800743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018064.800744:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018064.800746:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2331 00000100:00080000:0.0:1734018064.800747:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018064.800749:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018064.800751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018064.800753:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018064/real 1734018064] req@ffff8800a934dc00 x1818247334392320/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018064.800755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334392960/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018064.800759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018064.800761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018064.800762:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018064.800764:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2331 00000100:00080000:0.0:1734018064.800765:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018064.800765:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334392320/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018064.800767:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018064.800769:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018064.800770:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018064.800772:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018064.800773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334393088/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018064.800775:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334392320:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018064.800779:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334392448:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018064.800788:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018064.800792:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018064/real 1734018064] req@ffff8800a934d880 x1818247334392448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018064.800802:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334392448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018064.800806:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018064.800807:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018064.800809:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334392448:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018064.800822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334392576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018064.800830:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018064.800833:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018064/real 1734018064] req@ffff8800a934df80 x1818247334392576/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018064.800838:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334392576/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018064.800842:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018064.800843:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018064.800845:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334392576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018064.800848:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334392704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018064.800854:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018064.800857:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018064/real 1734018064] req@ffff8800a934d500 x1818247334392704/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018064.800862:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334392704/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018064.800866:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018064.800867:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018064.800869:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334392704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018064.800872:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334392832:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018064.800879:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018064.800882:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018064/real 1734018064] req@ffff8800a934fb80 x1818247334392832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018064.800887:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334392832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018064.800890:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018064.800892:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018064.800893:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334392832:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018064.800896:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334392960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018064.800902:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018064.800905:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018064/real 1734018064] req@ffff8800a934f100 x1818247334392960/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018064.800911:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334392960/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018064.800914:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018064.800915:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018064.800917:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334392960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018064.800920:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334393088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018064.800934:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018064.800937:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018064/real 1734018064] req@ffff8800a934e300 x1818247334393088/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018064.800943:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334393088/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018064.800947:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018064.800948:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018064.800949:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334393088:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018065.474772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018066.476833:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018067.479167:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018068.480670:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018069.482715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018069.808647:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018069.808659:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018069.808663:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018069.808675:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2336 00000100:00080000:0.0:1734018069.808685:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018069.808693:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018069.808731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334393216/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018069.808767:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018069.808771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018069.808773:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018069.808777:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2336 00000100:00080000:0.0:1734018069.808781:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018069.808784:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018069.808795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018069.808806:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334393344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018069.808816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018069.808819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018069.808821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018069.808825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2336 00000100:00080000:0.0:1734018069.808827:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018069.808831:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018069.808837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018069.808847:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334393472/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018069.808856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018069.808859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018069.808861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018069.808864:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2336 00000100:00080000:0.0:1734018069.808866:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018069.808869:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018069.808873:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018069.808881:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334393600/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018069.808890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018069.808890:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334393216:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018069.808893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018069.808894:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018069.808898:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2336 00000100:00080000:0.0:1734018069.808900:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018069.808903:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018069.808907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018069.808914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334393728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018069.808922:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018069.808925:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018069.808927:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018069.808931:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2336 00000100:00080000:0.0:1734018069.808942:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018069.808946:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018069.808950:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018069.808958:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018069.808959:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334393856/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018069.808967:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018069.808970:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018069.808971:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018069.808974:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2336 00000100:00080000:0.0:1734018069.808977:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018069.808980:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018069.808980:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018069/real 1734018069] req@ffff8800a934ce00 x1818247334393216/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018069.808984:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018069.808992:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334393984/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018069.809000:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334393216/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018069.809011:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018069.809014:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018069.809019:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334393216:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018069.809026:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334393344:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018069.809043:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018069.809049:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018069/real 1734018069] req@ffff8800a934e680 x1818247334393344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018069.809061:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334393344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018069.809069:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018069.809072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018069.809075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334393344:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018069.809081:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334393472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018069.809094:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018069.809100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018069/real 1734018069] req@ffff8800a934e300 x1818247334393472/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018069.809110:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334393472/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018069.809118:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018069.809120:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018069.809123:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334393472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018069.809147:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334393600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018069.809161:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018069.809167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018069/real 1734018069] req@ffff8800a934f100 x1818247334393600/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018069.809177:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334393600/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018069.809185:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018069.809187:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018069.809190:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334393600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018069.809196:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334393728:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018069.809208:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018069.809214:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018069/real 1734018069] req@ffff8800a934fb80 x1818247334393728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018069.809225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334393728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018069.809232:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018069.809235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018069.809237:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334393728:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018069.809243:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334393856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018069.809255:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018069.809261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018069/real 1734018069] req@ffff8800a934d500 x1818247334393856/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018069.809271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334393856/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018069.809278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018069.809280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018069.809283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334393856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018069.809288:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334393984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018069.809301:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018069.809307:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018069/real 1734018069] req@ffff8800a934df80 x1818247334393984/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018069.809317:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334393984/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018069.809325:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018069.809327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018069.809330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334393984:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018070.484753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018071.486704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018072.488721:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018073.490733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018074.492784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018074.816605:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018074.816616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018074.816620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018074.816632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2341 00000100:00080000:0.0:1734018074.816640:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018074.816648:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018074.816684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334394112/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018074.816724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018074.816729:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018074.816731:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018074.816735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2341 00000100:00080000:0.0:1734018074.816738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018074.816742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018074.816752:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018074.816763:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334394240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018074.816772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018074.816775:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018074.816777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018074.816781:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2341 00000100:00080000:0.0:1734018074.816784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018074.816788:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018074.816792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018074.816801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334394368/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018074.816809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018074.816812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018074.816814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018074.816817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2341 00000100:00080000:0.0:1734018074.816820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018074.816824:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018074.816828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018074.816834:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334394112:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018074.816836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334394496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018074.816845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018074.816848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018074.816849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018074.816853:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2341 00000100:00080000:0.0:1734018074.816855:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018074.816858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018074.816862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018074.816870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334394624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018074.816878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018074.816881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018074.816882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018074.816885:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2341 00000100:00080000:0.0:1734018074.816888:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018074.816891:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018074.816895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018074.816903:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018074.816904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334394752/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018074.816912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018074.816915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018074.816917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018074.816920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2341 00000100:00080000:0.0:1734018074.816923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018074.816925:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018074/real 1734018074] req@ffff8800a934d880 x1818247334394112/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018074.816927:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018074.816931:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018074.816938:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334394880/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018074.816944:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334394112/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018074.816955:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018074.816958:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018074.816963:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334394112:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018074.816970:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334394240:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018074.816987:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018074.816994:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018074/real 1734018074] req@ffff8800a934dc00 x1818247334394240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018074.817005:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334394240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018074.817012:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018074.817016:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018074.817019:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334394240:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018074.817026:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334394368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018074.817040:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018074.817046:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018074/real 1734018074] req@ffff8800a934ea00 x1818247334394368/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018074.817056:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334394368/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018074.817065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018074.817067:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018074.817071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334394368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018074.817093:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334394496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018074.817106:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018074.817112:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018074/real 1734018074] req@ffff8800a934c380 x1818247334394496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018074.817123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334394496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018074.817131:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018074.817133:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018074.817136:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334394496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018074.817141:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334394624:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018074.817157:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018074.817163:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018074/real 1734018074] req@ffff8800a934ca80 x1818247334394624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018074.817173:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334394624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018074.817180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018074.817183:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018074.817186:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334394624:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018074.817191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334394752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018074.817202:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018074.817209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018074/real 1734018074] req@ffff8800a934ed80 x1818247334394752/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018074.817219:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334394752/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018074.817226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018074.817228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018074.817231:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334394752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018074.817236:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334394880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018074.817280:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018074.817289:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018074/real 1734018074] req@ffff8800a934c700 x1818247334394880/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018074.817306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334394880/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018074.817314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018074.817316:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018074.817319:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334394880:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018075.494708:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018076.496734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018077.498787:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018078.500769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018079.502829:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018079.824610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018079.824620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018079.824625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018079.824635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2346 00000100:00080000:0.0:1734018079.824644:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018079.824652:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018079.824688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334395008/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018079.824724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018079.824728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018079.824730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018079.824734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2346 00000100:00080000:0.0:1734018079.824737:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018079.824741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018079.824751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018079.824762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334395136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018079.824772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018079.824776:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018079.824777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018079.824781:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2346 00000100:00080000:0.0:1734018079.824784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018079.824787:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018079.824792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018079.824802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334395264/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018079.824811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018079.824815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018079.824816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018079.824819:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2346 00000100:00080000:0.0:1734018079.824822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018079.824825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018079.824829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018079.824837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334395392/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018079.824846:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018079.824849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018079.824851:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018079.824854:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2346 00000100:00080000:0.0:1734018079.824857:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018079.824858:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334395008:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018079.824861:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018079.824865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018079.824873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334395520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018079.824881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018079.824884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018079.824885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018079.824889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2346 00000100:00080000:0.0:1734018079.824891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018079.824894:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018079.824898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018079.824906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334395648/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018079.824914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018079.824917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018079.824918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018079.824921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2346 00000100:00080000:0.0:1734018079.824934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018079.824937:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018079.824942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018079.824950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334395776/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018079.824951:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018079.824979:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018079/real 1734018079] req@ffff8800a934d180 x1818247334395008/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018079.825001:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334395008/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018079.825014:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018079.825018:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018079.825024:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334395008:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018079.825032:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334395136:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018079.825050:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018079.825057:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018079/real 1734018079] req@ffff8800a934c000 x1818247334395136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018079.825069:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334395136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018079.825077:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018079.825081:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018079.825084:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334395136:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018079.825091:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334395264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018079.825106:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018079.825113:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018079/real 1734018079] req@ffff8800a934c700 x1818247334395264/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018079.825123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334395264/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018079.825131:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018079.825134:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018079.825138:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334395264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018079.825205:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334395392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018079.825224:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018079.825232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018079/real 1734018079] req@ffff8800a934ed80 x1818247334395392/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018079.825243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334395392/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018079.825252:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018079.825255:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018079.825258:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334395392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018079.825264:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334395520:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018079.825281:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018079.825287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018079/real 1734018079] req@ffff8800a934ca80 x1818247334395520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018079.825299:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334395520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018079.825307:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018079.825309:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018079.825312:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334395520:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018079.825318:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334395648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018079.825331:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018079.825337:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018079/real 1734018079] req@ffff8800a934c380 x1818247334395648/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018079.825347:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334395648/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018079.825354:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018079.825356:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018079.825359:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334395648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018079.825365:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334395776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018079.825379:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018079.825386:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018079/real 1734018079] req@ffff8800a934ea00 x1818247334395776/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018079.825403:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334395776/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018079.825411:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018079.825413:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018079.825416:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334395776:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018080.504695:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018081.506724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018082.508737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018083.510791:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018084.512741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018084.832599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018084.832608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018084.832613:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018084.832624:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2351 00000100:00080000:0.0:1734018084.832632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018084.832640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018084.832677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334395904/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018084.832713:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018084.832717:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018084.832719:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018084.832723:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2351 00000100:00080000:0.0:1734018084.832726:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018084.832731:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018084.832740:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018084.832752:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334396032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018084.832762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018084.832765:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018084.832767:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018084.832771:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2351 00000100:00080000:0.0:1734018084.832773:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018084.832777:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018084.832783:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018084.832791:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334396160/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018084.832800:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018084.832803:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018084.832805:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018084.832808:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2351 00000100:00080000:0.0:1734018084.832810:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018084.832814:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018084.832818:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018084.832825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334396288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018084.832834:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018084.832837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018084.832839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018084.832842:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2351 00000100:00080000:0.0:1734018084.832845:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018084.832848:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018084.832852:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018084.832852:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334395904:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018084.832860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334396416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018084.832869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018084.832871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018084.832873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018084.832877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2351 00000100:00080000:0.0:1734018084.832880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018084.832883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018084.832887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018084.832896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334396544/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018084.832904:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018084.832907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018084.832908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018084.832912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2351 00000100:00080000:0.0:1734018084.832932:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018084.832935:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018084.832939:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018084.832947:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334396672/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018084.832951:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018084.832979:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018084/real 1734018084] req@ffff8800a934dc00 x1818247334395904/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018084.833002:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334395904/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018084.833014:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018084.833018:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018084.833023:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334395904:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018084.833031:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334396032:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018084.833051:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018084.833058:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018084/real 1734018084] req@ffff8800a934d880 x1818247334396032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018084.833069:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334396032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018084.833122:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018084.833127:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018084.833132:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334396032:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018084.833140:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334396160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018084.833160:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018084.833168:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018084/real 1734018084] req@ffff8800a934df80 x1818247334396160/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018084.833180:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334396160/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018084.833188:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018084.833191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018084.833194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334396160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018084.833226:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334396288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018084.833240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018084.833246:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018084/real 1734018084] req@ffff8800a934d500 x1818247334396288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018084.833256:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334396288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018084.833265:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018084.833268:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018084.833271:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334396288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018084.833277:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334396416:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018084.833295:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018084.833301:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018084/real 1734018084] req@ffff8800a934fb80 x1818247334396416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018084.833312:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334396416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018084.833320:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018084.833322:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018084.833325:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334396416:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018084.833331:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334396544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018084.833343:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018084.833349:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018084/real 1734018084] req@ffff8800a934f100 x1818247334396544/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018084.833360:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334396544/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018084.833368:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018084.833371:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018084.833374:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334396544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018084.833380:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334396672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018084.833394:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018084.833400:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018084/real 1734018084] req@ffff8800a934e300 x1818247334396672/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018084.833416:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334396672/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018084.833423:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018084.833426:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018084.833429:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334396672:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018085.514743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018086.516752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018087.518736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018088.520760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018089.522729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018089.840596:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018089.840606:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018089.840611:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018089.840623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2356 00000100:00080000:0.0:1734018089.840631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018089.840639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018089.840677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334396800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018089.840721:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018089.840725:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018089.840727:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018089.840731:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2356 00000100:00080000:0.0:1734018089.840735:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018089.840738:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018089.840749:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018089.840761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334396928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018089.840770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018089.840773:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018089.840774:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018089.840778:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2356 00000100:00080000:0.0:1734018089.840781:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018089.840785:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018089.840789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018089.840799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334397056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018089.840808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018089.840811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018089.840813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018089.840817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2356 00000100:00080000:0.0:1734018089.840819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018089.840823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018089.840827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018089.840835:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334397184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018089.840844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018089.840847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018089.840849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018089.840853:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2356 00000100:00080000:0.0:1734018089.840855:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018089.840858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018089.840863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018089.840864:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334396800:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018089.840871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334397312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018089.840878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018089.840881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018089.840882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018089.840886:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2356 00000100:00080000:0.0:1734018089.840889:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018089.840892:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018089.840896:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018089.840904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334397440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018089.840912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018089.840915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018089.840917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018089.840920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2356 00000100:00080000:0.0:1734018089.840923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018089.840926:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018089.840930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018089.840938:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334397568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018089.840963:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018089.840992:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018089/real 1734018089] req@ffff8800a934e680 x1818247334396800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018089.841054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334396800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018089.841067:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018089.841071:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018089.841077:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334396800:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018089.841085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334396928:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018089.841108:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018089.841116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018089/real 1734018089] req@ffff8800a934ce00 x1818247334396928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018089.841128:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334396928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018089.841136:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018089.841140:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018089.841143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334396928:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018089.841151:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334397056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018089.841166:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018089.841172:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018089/real 1734018089] req@ffff8800a934e300 x1818247334397056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018089.841182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334397056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018089.841191:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018089.841193:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018089.841197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334397056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018089.841228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334397184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018089.841242:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018089.841248:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018089/real 1734018089] req@ffff8800a934f100 x1818247334397184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018089.841259:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334397184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018089.841268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018089.841270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018089.841273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334397184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018089.841279:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334397312:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018089.841296:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018089.841303:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018089/real 1734018089] req@ffff8800a934fb80 x1818247334397312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018089.841314:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334397312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018089.841321:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018089.841324:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018089.841326:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334397312:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018089.841333:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334397440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018089.841345:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018089.841351:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018089/real 1734018089] req@ffff8800a934d500 x1818247334397440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018089.841361:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334397440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018089.841368:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018089.841370:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018089.841373:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334397440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018089.841379:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334397568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018089.841393:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018089.841399:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018089/real 1734018089] req@ffff8800a934df80 x1818247334397568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018089.841417:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334397568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018089.841425:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018089.841427:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018089.841430:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334397568:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018090.524751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018091.526737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018092.528733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018093.530695:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018094.532783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018094.848623:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018094.848633:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018094.848637:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018094.848648:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2361 00000100:00080000:0.0:1734018094.848657:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018094.848665:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018094.848702:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334397696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018094.848739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018094.848743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018094.848745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018094.848749:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2361 00000100:00080000:0.0:1734018094.848753:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018094.848756:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018094.848767:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018094.848778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334397824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018094.848788:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018094.848792:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018094.848793:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018094.848797:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2361 00000100:00080000:0.0:1734018094.848800:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018094.848804:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018094.848808:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018094.848817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334397952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018094.848825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018094.848828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018094.848830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018094.848833:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2361 00000100:00080000:0.0:1734018094.848836:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018094.848839:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018094.848843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018094.848852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334398080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018094.848860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018094.848863:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018094.848865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018094.848868:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2361 00000100:00080000:0.0:1734018094.848871:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018094.848874:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018094.848878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018094.848886:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334398208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018094.848895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018094.848898:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018094.848900:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018094.848903:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2361 00000100:00080000:0.0:1734018094.848905:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018094.848943:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018094.848948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018094.848950:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334397696:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018094.848959:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334398336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018094.848968:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018094.848971:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018094.848972:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018094.848976:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2361 00000100:00080000:0.0:1734018094.848979:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018094.848982:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018094.848986:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018094.848993:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334398464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018094.849052:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018094.849082:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018094/real 1734018094] req@ffff8800a934d880 x1818247334397696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018094.849104:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334397696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018094.849117:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018094.849121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018094.849126:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334397696:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018094.849135:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334397824:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018094.849155:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018094.849162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018094/real 1734018094] req@ffff8800a934dc00 x1818247334397824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018094.849173:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334397824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018094.849181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018094.849186:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018094.849190:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334397824:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018094.849197:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334397952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018094.849212:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018094.849218:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018094/real 1734018094] req@ffff8800a934ea00 x1818247334397952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018094.849228:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334397952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018094.849236:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018094.849239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018094.849243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334397952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018094.849249:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334398080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018094.849262:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018094.849269:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018094/real 1734018094] req@ffff8800a934c380 x1818247334398080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018094.849279:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334398080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018094.849286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018094.849288:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018094.849292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334398080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018094.849326:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334398208:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018094.849343:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018094.849349:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018094/real 1734018094] req@ffff8800a934ca80 x1818247334398208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018094.849361:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334398208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018094.849368:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018094.849370:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018094.849373:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334398208:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018094.849379:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334398336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018094.849392:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018094.849398:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018094/real 1734018094] req@ffff8800a934ed80 x1818247334398336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018094.849408:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334398336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018094.849415:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018094.849418:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018094.849421:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334398336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018094.849427:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334398464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018094.849439:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018094.849445:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018094/real 1734018094] req@ffff8800a934c700 x1818247334398464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018094.849461:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334398464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018094.849558:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018094.849561:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018094.849564:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334398464:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018095.534741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018096.536710:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018097.538782:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018098.540744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018099.542730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018099.856610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018099.856620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018099.856625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018099.856634:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2366 00000100:00080000:0.0:1734018099.856644:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018099.856652:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018099.856688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334398592/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018099.856724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018099.856728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018099.856730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018099.856735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2366 00000100:00080000:0.0:1734018099.856738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018099.856742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018099.856753:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018099.856764:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334398720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018099.856774:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018099.856777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018099.856778:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018099.856782:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2366 00000100:00080000:0.0:1734018099.856785:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018099.856788:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018099.856794:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018099.856805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334398848/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018099.856813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018099.856817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018099.856819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018099.856848:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2366 00000100:00080000:0.0:1734018099.856851:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018099.856855:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018099.856859:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018099.856868:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334398976/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018099.856877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018099.856880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018099.856882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018099.856885:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2366 00000100:00080000:0.0:1734018099.856888:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018099.856891:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018099.856895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018099.856902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334399104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018099.856912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018099.856915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018099.856917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018099.856917:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334398592:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018099.856921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2366 00000100:00080000:0.0:1734018099.856924:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018099.856927:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018099.856932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018099.856940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334399232/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018099.856948:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018099.856951:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018099.856952:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018099.856955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2366 00000100:00080000:0.0:1734018099.856958:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018099.856961:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018099.856965:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018099.856972:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334399360/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018099.857017:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018099.857047:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018099/real 1734018099] req@ffff8800a934c000 x1818247334398592/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018099.857069:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334398592/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018099.857081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018099.857085:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018099.857090:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334398592:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018099.857098:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334398720:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018099.857117:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018099.857124:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018099/real 1734018099] req@ffff8800a934d180 x1818247334398720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018099.857136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334398720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018099.857144:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018099.857148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018099.857151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334398720:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018099.857158:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334398848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018099.857174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018099.857180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018099/real 1734018099] req@ffff8800a934c700 x1818247334398848/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018099.857190:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334398848/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018099.857199:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018099.857201:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018099.857205:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334398848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018099.857211:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334398976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018099.857225:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018099.857231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018099/real 1734018099] req@ffff8800a934ed80 x1818247334398976/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018099.857240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334398976/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018099.857249:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018099.857251:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018099.857255:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334398976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018099.857288:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334399104:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018099.857305:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018099.857312:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018099/real 1734018099] req@ffff8800a934ca80 x1818247334399104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018099.857323:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334399104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018099.857330:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018099.857332:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018099.857335:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334399104:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018099.857341:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334399232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018099.857354:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018099.857360:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018099/real 1734018099] req@ffff8800a934c380 x1818247334399232/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018099.857369:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334399232/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018099.857377:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018099.857379:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018099.857382:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334399232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018099.857388:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334399360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018099.857401:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018099.857407:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018099/real 1734018099] req@ffff8800a934ea00 x1818247334399360/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018099.857424:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334399360/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018099.857433:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018099.857435:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018099.857438:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334399360:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018100.544721:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018101.546771:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018102.548728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018103.550815:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018104.552726:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018104.864610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018104.864619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018104.864624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018104.864635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2371 00000100:00080000:0.0:1734018104.864644:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018104.864652:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018104.864690:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334399488/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018104.864734:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018104.864774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018104.864776:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018104.864781:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2371 00000100:00080000:0.0:1734018104.864785:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018104.864789:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018104.864799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018104.864812:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334399616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018104.864822:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018104.864826:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018104.864827:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018104.864831:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2371 00000100:00080000:0.0:1734018104.864834:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018104.864837:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018104.864842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018104.864850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334399744/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018104.864860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018104.864863:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018104.864865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018104.864868:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2371 00000100:00080000:0.0:1734018104.864871:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018104.864875:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018104.864879:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018104.864880:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334399488:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018104.864887:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334399872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018104.864896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018104.864899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018104.864900:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018104.864903:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2371 00000100:00080000:0.0:1734018104.864906:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018104.864909:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018104.864913:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018104.864921:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334400000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018104.864929:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018104.864932:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018104.864933:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018104.864937:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2371 00000100:00080000:0.0:1734018104.864939:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018104.864942:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018104.864946:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018104.864956:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334400128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018104.864964:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018104.864967:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018104.864969:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018104.864972:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2371 00000100:00080000:0.0:1734018104.864976:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018104.864979:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018104.864979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018104.864983:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018104.864991:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334400256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018104.865009:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018104/real 1734018104] req@ffff8800a934dc00 x1818247334399488/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018104.865032:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334399488/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018104.865046:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018104.865049:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018104.865055:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334399488:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018104.865064:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334399616:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018104.865083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018104.865090:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018104/real 1734018104] req@ffff8800a934d880 x1818247334399616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018104.865101:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334399616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018104.865110:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018104.865114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018104.865117:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334399616:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018104.865147:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334399744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018104.865162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018104.865169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018104/real 1734018104] req@ffff8800a934df80 x1818247334399744/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018104.865180:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334399744/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018104.865188:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018104.865190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018104.865193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334399744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018104.865200:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334399872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018104.865213:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018104.865219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018104/real 1734018104] req@ffff8800a934d500 x1818247334399872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018104.865229:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334399872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018104.865236:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018104.865239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018104.865242:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334399872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018104.865248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334400000:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018104.865266:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018104.865272:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018104/real 1734018104] req@ffff8800a934fb80 x1818247334400000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018104.865283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334400000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018104.865290:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018104.865293:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018104.865296:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334400000:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018104.865301:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334400128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018104.865314:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018104.865320:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018104/real 1734018104] req@ffff8800a934f100 x1818247334400128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018104.865330:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334400128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018104.865337:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018104.865339:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018104.865342:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334400128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018104.865348:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334400256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018104.865361:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018104.865367:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018104/real 1734018104] req@ffff8800a934e300 x1818247334400256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018104.865384:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334400256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018104.865391:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018104.865393:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018104.865396:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334400256:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018105.554785:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018106.556752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018107.558768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018108.560728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018109.562808:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018109.872610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018109.872620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018109.872625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018109.872635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2376 00000100:00080000:0.0:1734018109.872645:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018109.872693:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018109.872731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334400384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018109.872765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018109.872769:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018109.872771:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018109.872776:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2376 00000100:00080000:0.0:1734018109.872779:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018109.872783:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018109.872794:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018109.872805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334400512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018109.872816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018109.872819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018109.872821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018109.872824:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2376 00000100:00080000:0.0:1734018109.872827:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018109.872830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018109.872835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018109.872845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334400640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018109.872854:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018109.872857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018109.872859:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018109.872862:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2376 00000100:00080000:0.0:1734018109.872865:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018109.872868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018109.872872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018109.872880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334400768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018109.872888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018109.872891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018109.872893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018109.872896:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2376 00000100:00080000:0.0:1734018109.872899:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018109.872902:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018109.872906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018109.872913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334400896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018109.872923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018109.872926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018109.872927:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018109.872930:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2376 00000100:00080000:0.0:1734018109.872933:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018109.872936:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018109.872938:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334400384:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018109.872940:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018109.872948:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334401024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018109.872956:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018109.872959:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018109.872961:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018109.872964:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2376 00000100:00080000:0.0:1734018109.872966:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018109.872969:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018109.872973:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018109.872981:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334401152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018109.873040:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018109.873069:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018109/real 1734018109] req@ffff8800a934ce00 x1818247334400384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018109.873092:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334400384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018109.873105:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018109.873109:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018109.873114:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334400384:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018109.873123:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334400512:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018109.873142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018109.873149:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018109/real 1734018109] req@ffff8800a934e680 x1818247334400512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018109.873161:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334400512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018109.873169:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018109.873173:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018109.873176:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334400512:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018109.873184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334400640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018109.873198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018109.873205:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018109/real 1734018109] req@ffff8800a934e300 x1818247334400640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018109.873215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334400640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018109.873223:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018109.873226:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018109.873230:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334400640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018109.873236:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334400768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018109.873250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018109.873256:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018109/real 1734018109] req@ffff8800a934f100 x1818247334400768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018109.873265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334400768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018109.873273:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018109.873275:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018109.873278:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334400768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018109.873309:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334400896:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018109.873327:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018109.873333:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018109/real 1734018109] req@ffff8800a934fb80 x1818247334400896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018109.873344:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334400896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018109.873352:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018109.873355:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018109.873358:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334400896:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018109.873363:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334401024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018109.873376:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018109.873382:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018109/real 1734018109] req@ffff8800a934d500 x1818247334401024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018109.873392:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334401024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018109.873399:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018109.873401:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018109.873404:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334401024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018109.873410:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334401152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018109.873423:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018109.873429:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018109/real 1734018109] req@ffff8800a934df80 x1818247334401152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018109.873446:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334401152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018109.873454:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018109.873456:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018109.873459:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334401152:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018110.564754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018111.566703:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018112.568777:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018113.570743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018114.572731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018114.880645:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018114.880655:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018114.880660:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018114.880670:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2381 00000100:00080000:0.0:1734018114.880679:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018114.880687:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018114.880724:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334401280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018114.880758:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018114.880762:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018114.880764:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018114.880769:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2381 00000100:00080000:0.0:1734018114.880772:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018114.880776:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018114.880786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018114.880797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334401408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018114.880807:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018114.880810:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018114.880811:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018114.880815:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2381 00000100:00080000:0.0:1734018114.880818:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018114.880821:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018114.880827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018114.880835:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334401536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018114.880844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018114.880847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018114.880849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018114.880852:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2381 00000100:00080000:0.0:1734018114.880854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018114.880858:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018114.880862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018114.880869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334401664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018114.880877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018114.880880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018114.880882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018114.880885:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2381 00000100:00080000:0.0:1734018114.880887:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018114.880890:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018114.880894:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018114.880901:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334401792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018114.880911:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018114.880913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018114.880915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018114.880919:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2381 00000100:00080000:0.0:1734018114.880922:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018114.880925:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018114.880929:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018114.880934:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334401280:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018114.880939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334401920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018114.880947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018114.880950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018114.880951:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018114.880955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2381 00000100:00080000:0.0:1734018114.880957:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018114.880960:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018114.880964:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018114.880972:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334402048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018114.881034:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018114.881063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018114/real 1734018114] req@ffff8800a934d880 x1818247334401280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018114.881086:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334401280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018114.881099:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018114.881102:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018114.881108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334401280:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018114.881116:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334401408:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018114.881135:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018114.881142:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018114/real 1734018114] req@ffff8800a934dc00 x1818247334401408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018114.881154:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334401408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018114.881162:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018114.881166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018114.881170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334401408:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018114.881177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334401536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018114.881192:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018114.881198:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018114/real 1734018114] req@ffff8800a934ea00 x1818247334401536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018114.881209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334401536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018114.881217:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018114.881220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018114.881224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334401536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018114.881230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334401664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018114.881244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018114.881250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018114/real 1734018114] req@ffff8800a934c380 x1818247334401664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018114.881260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334401664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018114.881268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018114.881270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018114.881274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334401664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018114.881307:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334401792:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018114.881325:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018114.881331:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018114/real 1734018114] req@ffff8800a934ca80 x1818247334401792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018114.881343:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334401792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018114.881350:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018114.881353:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018114.881355:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334401792:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018114.881362:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334401920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018114.881374:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018114.881380:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018114/real 1734018114] req@ffff8800a934ed80 x1818247334401920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018114.881390:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334401920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018114.881397:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018114.881399:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018114.881402:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334401920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018114.881408:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334402048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018114.881421:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018114.881427:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018114/real 1734018114] req@ffff8800a934c700 x1818247334402048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018114.881444:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334402048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018114.881452:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018114.881455:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018114.881458:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334402048:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018115.574742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018116.576780:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018117.578752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018118.580778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018119.582798:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018119.888626:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018119.888639:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018119.888645:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018119.888660:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2386 00000100:00080000:0.0:1734018119.888671:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018119.888681:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018119.888729:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334402176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018119.888776:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018119.888781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018119.888783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018119.888788:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2386 00000100:00080000:0.0:1734018119.888791:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018119.888795:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018119.888806:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018119.888818:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334402304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018119.888828:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018119.888831:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018119.888833:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018119.888837:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2386 00000100:00080000:0.0:1734018119.888840:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018119.888844:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018119.888848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018119.888860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334402432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018119.888868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018119.888872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018119.888873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018119.888877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2386 00000100:00080000:0.0:1734018119.888879:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018119.888883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018119.888888:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018119.888897:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334402560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018119.888900:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334402176:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018119.888907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018119.888910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018119.888911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018119.888915:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2386 00000100:00080000:0.0:1734018119.888917:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018119.888921:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018119.888925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018119.888932:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334402688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018119.888940:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018119.888944:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018119.888945:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018119.888949:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2386 00000100:00080000:0.0:1734018119.888951:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018119.888955:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018119.888959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018119.888967:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334402816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018119.888975:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018119.888980:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018119.888981:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018119.888985:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2386 00000100:00080000:0.0:1734018119.888988:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018119.888991:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018119.888995:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018119.889003:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334402944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018119.889019:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334402304:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018119.889041:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334402432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018119.889068:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334402560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018119.889084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334402688:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018119.889099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334402816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018119.889114:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334402944:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018120.584682:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018121.586779:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018122.588742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018122.894738:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734018122.894785:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018122.894803:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018122.894841:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018122.894850:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018122.894857:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018122.894863:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018122.894869:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018122.894879:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734018122.895003:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018119/real 1734018122] req@ffff8800a934ed80 x1818247334402560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018122.895045:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334402560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018122.895059:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018122.895064:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018122.895070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334402560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018122.895083:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018119/real 1734018122] req@ffff8800a934ca80 x1818247334402688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018122.895103:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334402688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018122.895116:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018122.895121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018122.895127:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334402688:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018122.895139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018119/real 1734018122] req@ffff8800a934c380 x1818247334402816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018122.895157:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334402816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018122.895169:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018122.895173:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018122.895179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334402816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018122.895191:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018119/real 1734018122] req@ffff8800a934ea00 x1818247334402944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018122.895210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334402944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018122.895222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018122.895226:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018122.895232:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334402944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018122.895248:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018119/real 1734018122] req@ffff8800a934d180 x1818247334402176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018122.895270:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334402176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018122.895282:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018122.895286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018122.895292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334402176:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018122.895303:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018119/real 1734018122] req@ffff8800a934c000 x1818247334402304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018122.895322:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334402304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018122.895335:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018122.895340:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018122.895345:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334402304:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018122.895356:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018119/real 1734018122] req@ffff8800a934c700 x1818247334402432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018122.895374:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334402432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018122.895386:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018122.895391:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018122.895397:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334402432:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018123.590719:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018124.592743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018124.896619:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018124.896630:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018124.896634:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018124.896646:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2391 00000100:00080000:0.0:1734018124.896655:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018124.896663:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018124.896701:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334403072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018124.896737:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018124.896741:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018124.896743:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018124.896747:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2391 00000100:00080000:0.0:1734018124.896750:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018124.896754:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018124.896764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018124.896775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334403200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018124.896786:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018124.896789:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018124.896791:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018124.896795:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2391 00000100:00080000:0.0:1734018124.896797:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018124.896801:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018124.896806:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018124.896814:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334403328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018124.896823:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018124.896826:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018124.896828:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018124.896831:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2391 00000100:00080000:0.0:1734018124.896834:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018124.896837:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018124.896841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018124.896848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334403456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018124.896858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018124.896861:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018124.896863:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018124.896866:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2391 00000100:00080000:0.0:1734018124.896869:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018124.896872:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018124.896877:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018124.896879:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334403072:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018124.896885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334403584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018124.896893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018124.896896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018124.896897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018124.896900:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2391 00000100:00080000:0.0:1734018124.896903:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018124.896906:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018124.896910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018124.896919:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334403712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018124.896927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018124.896930:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018124.896931:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018124.896934:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2391 00000100:00080000:0.0:1734018124.896937:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018124.896940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018124.896944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018124.896952:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334403840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018124.896979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018124.897007:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018124/real 1734018124] req@ffff8800a934dc00 x1818247334403072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.897029:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334403072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.897042:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.897046:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018124.897051:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334403072:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018124.897059:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334403200:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018124.897078:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018124.897086:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018124/real 1734018124] req@ffff8800a934d880 x1818247334403200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.897097:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334403200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.897106:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.897108:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018124.897112:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334403200:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018124.897118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334403328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018124.897133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018124.897139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018124/real 1734018124] req@ffff8800a934df80 x1818247334403328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.897149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334403328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.897158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.897160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018124.897164:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334403328:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018124.897194:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334403456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018124.897208:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018124.897214:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018124/real 1734018124] req@ffff8800a934d500 x1818247334403456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.897225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334403456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.897234:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.897236:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018124.897239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334403456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018124.897246:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334403584:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018124.897259:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018124.897265:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018124/real 1734018124] req@ffff8800a934fb80 x1818247334403584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.897275:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334403584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.897282:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.897284:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018124.897287:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334403584:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018124.897294:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334403712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018124.897307:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018124.897314:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018124/real 1734018124] req@ffff8800a934f100 x1818247334403712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.897324:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334403712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.897331:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.897333:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018124.897336:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334403712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018124.897342:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334403840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018124.897355:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018124.897361:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018124/real 1734018124] req@ffff8800a934e300 x1818247334403840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.897370:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334403840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.897378:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.897380:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018124.897383:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334403840:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018125.594696:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018126.596763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018127.598763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018128.600743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018129.602756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018129.904619:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018129.904629:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018129.904633:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018129.904644:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2396 00000100:00080000:0.0:1734018129.904653:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018129.904660:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018129.904696:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334403968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018129.904731:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018129.904736:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018129.904738:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018129.904742:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2396 00000100:00080000:0.0:1734018129.904746:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018129.904750:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018129.904760:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018129.904771:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334404096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018129.904781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018129.904784:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018129.904786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018129.904790:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2396 00000100:00080000:0.0:1734018129.904793:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018129.904796:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018129.904801:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018129.904811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334404224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018129.904821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018129.904824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018129.904825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018129.904829:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2396 00000100:00080000:0.0:1734018129.904831:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018129.904835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018129.904839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018129.904846:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334404352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018129.904855:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018129.904859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018129.904860:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018129.904863:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2396 00000100:00080000:0.0:1734018129.904866:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018129.904869:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018129.904873:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018129.904880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334404480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018129.904890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018129.904893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018129.904894:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018129.904899:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2396 00000100:00080000:0.0:1734018129.904901:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018129.904904:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018129.904904:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334403968:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018129.904909:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018129.904917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334404608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018129.904925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018129.904928:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018129.904929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018129.904932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2396 00000100:00080000:0.0:1734018129.904935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018129.904938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018129.904942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018129.904949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334404736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018129.905006:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018129.905035:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018129/real 1734018129] req@ffff8800a934e680 x1818247334403968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018129.905059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334403968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018129.905073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018129.905077:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018129.905082:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334403968:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018129.905090:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334404096:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018129.905108:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018129.905115:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018129/real 1734018129] req@ffff8800a934ce00 x1818247334404096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018129.905127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334404096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018129.905135:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018129.905138:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018129.905142:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334404096:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018129.905148:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334404224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018129.905163:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018129.905169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018129/real 1734018129] req@ffff8800a934e300 x1818247334404224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018129.905179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334404224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018129.905187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018129.905190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018129.905193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334404224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018129.905200:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334404352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018129.905214:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018129.905220:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018129/real 1734018129] req@ffff8800a934f100 x1818247334404352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018129.905230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334404352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018129.905237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018129.905240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018129.905243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334404352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018129.905276:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334404480:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018129.905290:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018129.905297:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018129/real 1734018129] req@ffff8800a934fb80 x1818247334404480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018129.905307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334404480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018129.905347:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018129.905350:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018129.905353:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334404480:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018129.905360:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334404608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018129.905381:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018129.905389:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018129/real 1734018129] req@ffff8800a934d500 x1818247334404608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018129.905400:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334404608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018129.905408:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018129.905410:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018129.905413:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334404608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018129.905419:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334404736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018129.905433:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018129.905439:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018129/real 1734018129] req@ffff8800a934df80 x1818247334404736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018129.905449:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334404736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018129.905456:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018129.905458:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018129.905461:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334404736:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018130.604750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018131.606627:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018132.608731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018133.610750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018134.612740:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018134.912611:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018134.912621:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018134.912625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018134.912636:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2401 00000100:00080000:0.0:1734018134.912645:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018134.912653:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018134.912689:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334404864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018134.912730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018134.912734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018134.912736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018134.912740:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2401 00000100:00080000:0.0:1734018134.912744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018134.912747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018134.912758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018134.912769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334404992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018134.912778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018134.912781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018134.912783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018134.912787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2401 00000100:00080000:0.0:1734018134.912789:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018134.912793:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018134.912798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018134.912806:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334405120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018134.912815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018134.912818:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018134.912819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018134.912822:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2401 00000100:00080000:0.0:1734018134.912825:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018134.912828:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018134.912833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018134.912839:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334404864:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018134.912841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334405248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018134.912850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018134.912852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018134.912854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018134.912857:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2401 00000100:00080000:0.0:1734018134.912860:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018134.912863:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018134.912867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018134.912876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334405376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018134.912884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018134.912886:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018134.912888:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018134.912891:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2401 00000100:00080000:0.0:1734018134.912894:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018134.912897:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018134.912901:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018134.912908:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018134.912909:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334405504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018134.912917:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018134.912920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018134.912922:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018134.912925:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2401 00000100:00080000:0.0:1734018134.912928:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018134.912931:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018134.912931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018134/real 1734018134] req@ffff8800a934d880 x1818247334404864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018134.912935:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018134.912942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334405632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018134.912958:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334404864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018134.912969:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018134.912972:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018134.912976:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334404864:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018134.912984:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334404992:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018134.913001:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018134.913008:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018134/real 1734018134] req@ffff8800a934dc00 x1818247334404992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018134.913019:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334404992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018134.913026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018134.913029:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018134.913032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334404992:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018134.913038:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334405120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018134.913052:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018134.913058:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018134/real 1734018134] req@ffff8800a934c700 x1818247334405120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018134.913070:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334405120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018134.913082:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018134.913086:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018134.913092:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334405120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018134.913127:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334405248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018134.913156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018134.913169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018134/real 1734018134] req@ffff8800a934c000 x1818247334405248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018134.913187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334405248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018134.913201:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018134.913204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018134.913210:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334405248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018134.913220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334405376:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018134.913301:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018134.913311:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018134/real 1734018134] req@ffff8800a934d180 x1818247334405376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018134.913323:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334405376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018134.913335:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018134.913338:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018134.913344:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334405376:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018134.913353:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334405504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018134.913382:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018134.913395:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018134/real 1734018134] req@ffff8800a934ea00 x1818247334405504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018134.913412:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334405504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018134.913425:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018134.913428:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018134.913433:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334405504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018134.913441:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334405632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018134.913462:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018134.913517:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018134/real 1734018134] req@ffff8800a934c380 x1818247334405632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018134.913528:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334405632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018134.913536:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018134.913539:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018134.913542:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334405632:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018135.614700:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018136.616747:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018137.618720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018138.620712:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018139.622753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018139.920581:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018139.920591:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018139.920595:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018139.920606:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2406 00000100:00080000:0.0:1734018139.920615:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018139.920623:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018139.920659:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334405760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018139.920693:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018139.920697:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018139.920699:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018139.920703:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2406 00000100:00080000:0.0:1734018139.920706:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018139.920710:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018139.920720:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018139.920731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334405888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018139.920741:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018139.920745:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018139.920746:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018139.920750:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2406 00000100:00080000:0.0:1734018139.920753:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018139.920756:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018139.920761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018139.920771:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334406016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018139.920780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018139.920783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018139.920784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018139.920787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2406 00000100:00080000:0.0:1734018139.920790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018139.920793:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018139.920797:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018139.920798:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334405760:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018139.920805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334406144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018139.920814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018139.920817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018139.920819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018139.920822:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2406 00000100:00080000:0.0:1734018139.920825:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018139.920828:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018139.920832:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018139.920839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334406272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018139.920847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018139.920850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018139.920852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018139.920855:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2406 00000100:00080000:0.0:1734018139.920858:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018139.920861:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018139.920865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018139.920865:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018139.920872:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334406400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018139.920880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018139.920883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018139.920885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018139.920887:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018139/real 1734018139] req@ffff8800a934ca80 x1818247334405760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018139.920888:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2406 00000100:00080000:0.0:1734018139.920891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018139.920894:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018139.920898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018139.920905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334406528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018139.920914:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334405760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018139.920925:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018139.920928:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018139.920933:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334405760:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018139.920940:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334405888:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018139.920956:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018139.920963:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018139/real 1734018139] req@ffff8800a934ed80 x1818247334405888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018139.920974:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334405888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018139.920982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018139.920984:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018139.920987:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334405888:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018139.920993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334406016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018139.921006:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018139.921012:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018139/real 1734018139] req@ffff8800a934c380 x1818247334406016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018139.921022:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334406016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018139.921030:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018139.921032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018139.921036:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334406016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018139.921061:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334406144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018139.921075:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018139.921081:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018139/real 1734018139] req@ffff8800a934ea00 x1818247334406144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018139.921092:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334406144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018139.921099:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018139.921101:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018139.921104:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334406144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018139.921109:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334406272:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018139.921121:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018139.921127:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018139/real 1734018139] req@ffff8800a934d180 x1818247334406272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018139.921138:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334406272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018139.921180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018139.921182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018139.921186:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334406272:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018139.921193:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334406400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018139.921212:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018139.921219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018139/real 1734018139] req@ffff8800a934c000 x1818247334406400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018139.921230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334406400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018139.921238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018139.921240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018139.921243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334406400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018139.921248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334406528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018139.921261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018139.921267:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018139/real 1734018139] req@ffff8800a934c700 x1818247334406528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018139.921277:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334406528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018139.921284:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018139.921286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018139.921289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334406528:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018140.624757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018141.626689:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018142.628735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018143.630712:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018144.632749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018144.928599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018144.928609:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018144.928614:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018144.928624:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2411 00000100:00080000:0.0:1734018144.928632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018144.928640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018144.928677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334406656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018144.928711:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018144.928715:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018144.928717:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018144.928721:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2411 00000100:00080000:0.0:1734018144.928725:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018144.928729:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018144.928739:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018144.928750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334406784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018144.928760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018144.928763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018144.928765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018144.928769:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2411 00000100:00080000:0.0:1734018144.928772:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018144.928775:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018144.928781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018144.928789:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334406912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018144.928798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018144.928801:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018144.928803:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018144.928806:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2411 00000100:00080000:0.0:1734018144.928809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018144.928812:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018144.928816:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018144.928817:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334406656:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018144.928824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334407040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018144.928832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018144.928835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018144.928837:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018144.928840:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2411 00000100:00080000:0.0:1734018144.928842:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018144.928845:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018144.928850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018144.928857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334407168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018144.928865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018144.928868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018144.928870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018144.928874:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2411 00000100:00080000:0.0:1734018144.928877:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018144.928880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018144.928883:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018144.928884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018144.928894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334407296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018144.928902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018144.928905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018144.928905:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018144/real 1734018144] req@ffff8800a934dc00 x1818247334406656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018144.928906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018144.928909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2411 00000100:00080000:0.0:1734018144.928912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018144.928915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018144.928919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018144.928926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334407424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018144.928932:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334406656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018144.928942:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018144.928946:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018144.928950:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334406656:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018144.928957:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334406784:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018144.928974:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018144.928980:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018144/real 1734018144] req@ffff8800a934d880 x1818247334406784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018144.928991:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334406784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018144.928999:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018144.929001:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018144.929005:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334406784:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018144.929011:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334406912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018144.929024:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018144.929030:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018144/real 1734018144] req@ffff8800a934df80 x1818247334406912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018144.929040:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334406912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018144.929047:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018144.929051:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018144.929054:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334406912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018144.929111:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334407040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018144.929129:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018144.929136:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018144/real 1734018144] req@ffff8800a934d500 x1818247334407040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018144.929147:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334407040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018144.929155:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018144.929157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018144.929160:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334407040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018144.929165:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334407168:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018144.929178:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018144.929184:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018144/real 1734018144] req@ffff8800a934fb80 x1818247334407168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018144.929194:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334407168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018144.929202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018144.929204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018144.929207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334407168:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018144.929213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334407296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018144.929228:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018144.929234:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018144/real 1734018144] req@ffff8800a934f100 x1818247334407296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018144.929244:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334407296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018144.929251:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018144.929253:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018144.929256:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334407296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018144.929261:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334407424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018144.929273:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018144.929279:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018144/real 1734018144] req@ffff8800a934e300 x1818247334407424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018144.929289:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334407424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018144.929296:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018144.929298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018144.929301:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334407424:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018145.634738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018146.636750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018147.638734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018148.640741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018149.642755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018149.936627:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018149.936637:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018149.936641:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018149.936652:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2416 00000100:00080000:0.0:1734018149.936661:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018149.936668:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018149.936705:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334407552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018149.936747:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018149.936751:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018149.936753:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018149.936757:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2416 00000100:00080000:0.0:1734018149.936761:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018149.936764:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018149.936774:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018149.936785:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334407680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018149.936795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018149.936798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018149.936799:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018149.936803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2416 00000100:00080000:0.0:1734018149.936806:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018149.936810:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018149.936814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018149.936824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334407808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018149.936832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018149.936835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018149.936836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018149.936840:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2416 00000100:00080000:0.0:1734018149.936842:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018149.936846:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018149.936849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018149.936858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334407936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018149.936864:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334407552:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018149.936868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018149.936870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018149.936872:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018149.936875:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2416 00000100:00080000:0.0:1734018149.936877:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018149.936880:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018149.936884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018149.936891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334408064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018149.936899:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018149.936902:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018149.936904:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018149.936907:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2416 00000100:00080000:0.0:1734018149.936919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018149.936922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018149.936926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018149.936930:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018149.936935:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334408192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018149.936944:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018149.936947:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018149.936949:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018149.936951:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018149/real 1734018149] req@ffff8800a934ce00 x1818247334407552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018149.936952:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2416 00000100:00080000:0.0:1734018149.936955:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018149.936958:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018149.936962:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018149.937003:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334408320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018149.937020:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334407552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018149.937031:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018149.937035:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018149.937039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334407552:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018149.937047:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334407680:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018149.937067:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018149.937075:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018149/real 1734018149] req@ffff8800a934e680 x1818247334407680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018149.937086:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334407680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018149.937094:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018149.937096:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018149.937099:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334407680:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018149.937105:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334407808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018149.937119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018149.937125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018149/real 1734018149] req@ffff8800a934e300 x1818247334407808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018149.937136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334407808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018149.937143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018149.937146:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018149.937149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334407808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018149.937177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334407936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018149.937190:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018149.937196:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018149/real 1734018149] req@ffff8800a934f100 x1818247334407936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018149.937207:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334407936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018149.937214:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018149.937216:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018149.937219:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334407936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018149.937224:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334408064:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018149.937237:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018149.937243:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018149/real 1734018149] req@ffff8800a934fb80 x1818247334408064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018149.937252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334408064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018149.937260:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018149.937262:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018149.937265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334408064:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018149.937270:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334408192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018149.937286:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018149.937292:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018149/real 1734018149] req@ffff8800a934d500 x1818247334408192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018149.937302:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334408192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018149.937309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018149.937311:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018149.937314:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334408192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018149.937319:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334408320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018149.937332:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018149.937338:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018149/real 1734018149] req@ffff8800a934df80 x1818247334408320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018149.937347:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334408320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018149.937355:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018149.937357:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018149.937360:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334408320:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018150.644739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018151.646741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018152.648762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018153.650758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018154.652742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018154.944595:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018154.944605:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018154.944610:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018154.944621:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2421 00000100:00080000:0.0:1734018154.944631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018154.944639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018154.944675:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334408448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018154.944709:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018154.944713:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018154.944715:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018154.944720:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2421 00000100:00080000:0.0:1734018154.944723:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018154.944727:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018154.944737:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018154.944748:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334408576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018154.944758:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018154.944762:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018154.944763:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018154.944767:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2421 00000100:00080000:0.0:1734018154.944770:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018154.944774:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018154.944778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018154.944787:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334408704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018154.944795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018154.944798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018154.944800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018154.944803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2421 00000100:00080000:0.0:1734018154.944806:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018154.944809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018154.944813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018154.944818:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334408448:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018154.944821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334408832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018154.944830:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018154.944833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018154.944835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018154.944838:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2421 00000100:00080000:0.0:1734018154.944841:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018154.944844:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018154.944848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018154.944858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334408960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018154.944866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018154.944869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018154.944871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018154.944874:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2421 00000100:00080000:0.0:1734018154.944877:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018154.944880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018154.944917:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018154.944927:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018154.944928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334409088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018154.944937:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018154.944940:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018154.944941:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018154.944944:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2421 00000100:00080000:0.0:1734018154.944948:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018154.944951:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018154.944951:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018154/real 1734018154] req@ffff8800a934d880 x1818247334408448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018154.944955:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018154.944963:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334409216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018154.944979:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334408448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018154.944989:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018154.944992:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018154.944997:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334408448:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018154.945005:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334408576:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018154.945025:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018154.945032:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018154/real 1734018154] req@ffff8800a934dc00 x1818247334408576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018154.945043:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334408576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018154.945050:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018154.945053:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018154.945056:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334408576:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018154.945062:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334408704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018154.945076:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018154.945082:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018154/real 1734018154] req@ffff8800a934c700 x1818247334408704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018154.945092:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334408704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018154.945099:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018154.945103:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018154.945106:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334408704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018154.945129:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334408832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018154.945142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018154.945149:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018154/real 1734018154] req@ffff8800a934c000 x1818247334408832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018154.945159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334408832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018154.945167:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018154.945169:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018154.945172:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334408832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018154.945177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334408960:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018154.945189:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018154.945195:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018154/real 1734018154] req@ffff8800a934d180 x1818247334408960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018154.945205:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334408960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018154.945212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018154.945215:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018154.945218:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334408960:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018154.945224:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334409088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018154.945238:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018154.945245:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018154/real 1734018154] req@ffff8800a934ea00 x1818247334409088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018154.945255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334409088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018154.945262:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018154.945264:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018154.945267:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334409088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018154.945272:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334409216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018154.945284:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018154.945290:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018154/real 1734018154] req@ffff8800a934c380 x1818247334409216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018154.945300:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334409216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018154.945307:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018154.945309:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018154.945312:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334409216:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018155.654745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018156.656740:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018157.658718:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018158.660705:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018159.662770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018159.952607:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018159.952621:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018159.952627:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018159.952641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2426 00000100:00080000:0.0:1734018159.952653:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018159.952663:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018159.952711:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334409344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018159.952751:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018159.952755:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018159.952757:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018159.952763:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2426 00000100:00080000:0.0:1734018159.952766:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018159.952770:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018159.952781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018159.952793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334409472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018159.952847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018159.952851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018159.952853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018159.952858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2426 00000100:00080000:0.0:1734018159.952861:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018159.952865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018159.952871:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018159.952885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334409600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018159.952896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018159.952899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018159.952901:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018159.952905:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2426 00000100:00080000:0.0:1734018159.952907:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018159.952911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018159.952915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018159.952915:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334409344:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018159.952925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334409728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018159.952933:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018159.952936:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018159.952938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018159.952941:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2426 00000100:00080000:0.0:1734018159.952944:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018159.952948:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018159.952953:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018159.952961:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334409856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018159.952969:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018159.952972:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018159.952974:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018159.952979:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2426 00000100:00080000:0.0:1734018159.952982:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018159.952986:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018159.952990:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018159.952994:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018159.952999:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334409984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018159.953008:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018159.953011:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018159.953012:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018159.953016:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2426 00000100:00100000:1.0:1734018159.953017:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018159/real 1734018159] req@ffff8800a934ed80 x1818247334409344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018159.953019:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018159.953022:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018159.953026:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018159.953035:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334410112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018159.953045:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334409344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018159.953056:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018159.953060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018159.953065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334409344:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018159.953073:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334409472:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018159.953091:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018159.953098:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018159/real 1734018159] req@ffff8800a934ca80 x1818247334409472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018159.953109:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334409472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018159.953116:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018159.953119:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018159.953123:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334409472:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018159.953148:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334409600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018159.953162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018159.953168:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018159/real 1734018159] req@ffff8800a934c380 x1818247334409600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018159.953178:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334409600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018159.953186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018159.953190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018159.953193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334409600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018159.953200:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334409728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018159.953213:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018159.953219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018159/real 1734018159] req@ffff8800a934ea00 x1818247334409728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018159.953228:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334409728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018159.953236:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018159.953238:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018159.953241:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334409728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018159.953247:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334409856:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018159.953260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018159.953266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018159/real 1734018159] req@ffff8800a934d180 x1818247334409856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018159.953276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334409856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018159.953283:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018159.953285:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018159.953288:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334409856:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018159.953294:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334409984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018159.953310:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018159.953316:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018159/real 1734018159] req@ffff8800a934c000 x1818247334409984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018159.953327:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334409984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018159.953334:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018159.953336:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018159.953339:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334409984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018159.953345:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334410112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018159.953357:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018159.953363:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018159/real 1734018159] req@ffff8800a934c700 x1818247334410112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018159.953373:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334410112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018159.953380:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018159.953382:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018159.953385:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334410112:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018160.664745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018161.666728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018162.668730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018163.670704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018164.672690:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018164.960606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018164.960615:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018164.960620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018164.960630:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2431 00000100:00080000:0.0:1734018164.960640:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018164.960647:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018164.960684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334410240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018164.960756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018164.960761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018164.960763:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018164.960767:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2431 00000100:00080000:0.0:1734018164.960771:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018164.960775:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018164.960786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018164.960798:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334410368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018164.960808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018164.960811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018164.960813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018164.960817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2431 00000100:00080000:0.0:1734018164.960819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018164.960823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018164.960828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018164.960836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334410496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018164.960844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018164.960847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018164.960849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018164.960852:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2431 00000100:00080000:0.0:1734018164.960855:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018164.960859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018164.960863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018164.960867:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334410240:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018164.960871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334410624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018164.960880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018164.960883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018164.960885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018164.960888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2431 00000100:00080000:0.0:1734018164.960890:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018164.960893:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018164.960897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018164.960904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334410752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018164.960912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018164.960915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018164.960917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018164.960920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2431 00000100:00080000:0.0:1734018164.960923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018164.960926:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018164.960930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018164.960935:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018164.960939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334410880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018164.960947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018164.960951:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018164.960953:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018164.960956:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2431 00000100:00100000:1.0:1734018164.960957:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018164/real 1734018164] req@ffff8800a934dc00 x1818247334410240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018164.960959:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018164.960962:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018164.960966:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018164.960973:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334411008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018164.960985:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334410240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018164.960995:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018164.960998:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018164.961003:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334410240:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018164.961010:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334410368:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018164.961026:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018164.961033:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018164/real 1734018164] req@ffff8800a934d880 x1818247334410368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018164.961044:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334410368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018164.961052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018164.961054:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018164.961058:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334410368:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018164.961063:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334410496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018164.961077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018164.961083:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018164/real 1734018164] req@ffff8800a934df80 x1818247334410496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018164.961093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334410496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018164.961100:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018164.961104:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018164.961107:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334410496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018164.961136:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334410624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018164.961149:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018164.961155:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018164/real 1734018164] req@ffff8800a934d500 x1818247334410624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018164.961166:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334410624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018164.961173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018164.961175:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018164.961178:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334410624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018164.961184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334410752:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018164.961196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018164.961202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018164/real 1734018164] req@ffff8800a934fb80 x1818247334410752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018164.961212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334410752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018164.961219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018164.961221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018164.961224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334410752:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018164.961230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334410880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018164.961244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018164.961251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018164/real 1734018164] req@ffff8800a934f100 x1818247334410880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018164.961261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334410880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018164.961268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018164.961270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018164.961273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334410880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018164.961278:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334411008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018164.961290:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018164.961296:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018164/real 1734018164] req@ffff8800a934e300 x1818247334411008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018164.961306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334411008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018164.961313:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018164.961315:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018164.961318:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334411008:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018165.674764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018166.676761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018167.678763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018168.680732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018169.682704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018169.968601:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018169.968610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018169.968616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018169.968626:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2436 00000100:00080000:0.0:1734018169.968671:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018169.968680:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018169.968717:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334411136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018169.968750:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018169.968754:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018169.968757:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018169.968761:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2436 00000100:00080000:0.0:1734018169.968765:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018169.968769:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018169.968779:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018169.968790:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334411264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018169.968801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018169.968804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018169.968805:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018169.968809:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2436 00000100:00080000:0.0:1734018169.968812:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018169.968815:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018169.968820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018169.968830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334411392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018169.968838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018169.968841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018169.968843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018169.968846:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2436 00000100:00080000:0.0:1734018169.968849:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018169.968852:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018169.968856:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018169.968864:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334411520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018169.968867:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334411136:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018169.968874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018169.968877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018169.968878:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018169.968881:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2436 00000100:00080000:0.0:1734018169.968884:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018169.968888:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018169.968892:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018169.968900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334411648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018169.968907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018169.968910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018169.968912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018169.968915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2436 00000100:00080000:0.0:1734018169.968920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018169.968924:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018169.968927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018169.968935:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018169.968936:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334411776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018169.968943:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018169.968947:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018169.968948:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018169.968951:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2436 00000100:00080000:0.0:1734018169.968954:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018169.968957:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018169.968957:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018169/real 1734018169] req@ffff8800a934e680 x1818247334411136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018169.968961:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018169.968968:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334411904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018169.968985:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334411136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018169.968995:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018169.968998:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018169.969003:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334411136:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018169.969010:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334411264:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018169.969027:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018169.969033:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018169/real 1734018169] req@ffff8800a934ce00 x1818247334411264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018169.969045:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334411264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018169.969052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018169.969055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018169.969058:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334411264:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018169.969064:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334411392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018169.969077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018169.969084:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018169/real 1734018169] req@ffff8800a934e300 x1818247334411392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018169.969093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334411392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018169.969101:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018169.969103:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018169.969107:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334411392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018169.969132:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334411520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018169.969146:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018169.969152:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018169/real 1734018169] req@ffff8800a934f100 x1818247334411520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018169.969162:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334411520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018169.969170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018169.969172:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018169.969175:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334411520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018169.969180:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334411648:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018169.969192:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018169.969198:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018169/real 1734018169] req@ffff8800a934fb80 x1818247334411648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018169.969208:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334411648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018169.969215:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018169.969217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018169.969220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334411648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018169.969226:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334411776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018169.969241:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018169.969248:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018169/real 1734018169] req@ffff8800a934d500 x1818247334411776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018169.969258:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334411776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018169.969265:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018169.969267:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018169.969270:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334411776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018169.969276:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334411904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018169.969288:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018169.969294:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018169/real 1734018169] req@ffff8800a934df80 x1818247334411904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018169.969303:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334411904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018169.969310:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018169.969313:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018169.969316:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334411904:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018170.684718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018171.686761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018172.688741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018173.690772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018174.692768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018174.976629:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018174.976644:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018174.976648:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018174.976659:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2441 00000100:00080000:0.0:1734018174.976668:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018174.976676:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018174.976713:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334412032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018174.976746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018174.976750:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018174.976752:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018174.976757:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2441 00000100:00080000:0.0:1734018174.976760:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018174.976764:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018174.976774:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018174.976785:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334412160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018174.976795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018174.976798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018174.976800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018174.976804:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2441 00000100:00080000:0.0:1734018174.976807:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018174.976811:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018174.976817:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018174.976825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334412288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018174.976834:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018174.976837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018174.976839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018174.976842:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2441 00000100:00080000:0.0:1734018174.976844:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018174.976847:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018174.976851:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018174.976856:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334412032:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018174.976859:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334412416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018174.976868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018174.976871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018174.976873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018174.976876:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2441 00000100:00080000:0.0:1734018174.976878:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018174.976882:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018174.976885:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018174.976893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334412544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018174.976900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018174.976903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018174.976905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018174.976909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2441 00000100:00080000:0.0:1734018174.976912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018174.976915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018174.976919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018174.976923:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018174.976928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334412672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018174.976937:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018174.976940:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018174.976941:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018174.976944:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2441 00000100:00100000:1.0:1734018174.976945:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018174/real 1734018174] req@ffff8800a934d880 x1818247334412032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018174.976947:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018174.976950:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018174.976954:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018174.976961:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334412800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018174.976973:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334412032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018174.976983:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018174.976986:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018174.976991:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334412032:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018174.976998:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334412160:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018174.977015:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018174.977022:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018174/real 1734018174] req@ffff8800a934dc00 x1818247334412160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018174.977033:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334412160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018174.977041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018174.977043:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018174.977047:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334412160:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018174.977053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334412288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018174.977066:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018174.977072:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018174/real 1734018174] req@ffff8800a934c700 x1818247334412288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018174.977082:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334412288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018174.977089:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018174.977092:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018174.977095:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334412288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018174.977119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334412416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018174.977133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018174.977139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018174/real 1734018174] req@ffff8800a934c000 x1818247334412416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018174.977150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334412416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018174.977157:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018174.977159:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018174.977162:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334412416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018174.977167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334412544:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018174.977179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018174.977185:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018174/real 1734018174] req@ffff8800a934d180 x1818247334412544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018174.977195:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334412544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018174.977202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018174.977204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018174.977207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334412544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018174.977213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334412672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018174.977228:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018174.977234:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018174/real 1734018174] req@ffff8800a934ea00 x1818247334412672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018174.977244:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334412672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018174.977252:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018174.977254:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018174.977257:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334412672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018174.977262:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334412800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018174.977274:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018174.977280:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018174/real 1734018174] req@ffff8800a934c380 x1818247334412800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018174.977290:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334412800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018174.977297:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018174.977299:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018174.977302:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334412800:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018175.694808:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018176.696753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018177.698687:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018178.700756:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018179.702755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018179.984608:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018179.984618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018179.984623:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018179.984634:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2446 00000100:00080000:0.0:1734018179.984643:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018179.984651:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018179.984688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334412928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018179.984728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018179.984732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018179.984734:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018179.984738:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2446 00000100:00080000:0.0:1734018179.984741:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018179.984745:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018179.984756:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018179.984766:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334413056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018179.984776:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018179.984779:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018179.984780:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018179.984784:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2446 00000100:00080000:0.0:1734018179.984787:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018179.984791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018179.984796:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018179.984806:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334413184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018179.984814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018179.984817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018179.984819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018179.984822:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2446 00000100:00080000:0.0:1734018179.984824:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018179.984829:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018179.984832:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334412928:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018179.984834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018179.984842:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334413312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018179.984851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018179.984854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018179.984856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018179.984859:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2446 00000100:00080000:0.0:1734018179.984861:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018179.984865:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018179.984868:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018179.984876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334413440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018179.984884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018179.984887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018179.984888:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018179.984892:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2446 00000100:00080000:0.0:1734018179.984895:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018179.984898:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018179.984898:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018179.984902:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018179.984910:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334413568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018179.984918:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018179.984918:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018179/real 1734018179] req@ffff8800a934ca80 x1818247334412928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018179.984921:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018179.984923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018179.984927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2446 00000100:00080000:0.0:1734018179.984930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018179.984933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018179.984937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018179.984944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334413696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018179.984946:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334412928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018179.984956:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018179.984959:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018179.984964:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334412928:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018179.984971:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334413056:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018179.984988:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018179.984995:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018179/real 1734018179] req@ffff8800a934ed80 x1818247334413056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018179.985006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334413056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018179.985014:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018179.985016:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018179.985020:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334413056:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018179.985026:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334413184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018179.985039:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018179.985045:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018179/real 1734018179] req@ffff8800a934c380 x1818247334413184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018179.985055:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334413184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018179.985063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018179.985066:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018179.985069:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334413184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018179.985097:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334413312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018179.985110:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018179.985116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018179/real 1734018179] req@ffff8800a934ea00 x1818247334413312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018179.985127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334413312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018179.985134:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018179.985136:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018179.985139:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334413312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018179.985144:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334413440:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018179.985156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018179.985162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018179/real 1734018179] req@ffff8800a934d180 x1818247334413440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018179.985172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334413440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018179.985179:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018179.985181:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018179.985184:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334413440:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018179.985190:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334413568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018179.985205:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018179.985211:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018179/real 1734018179] req@ffff8800a934c000 x1818247334413568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018179.985221:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334413568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018179.985228:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018179.985230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018179.985233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334413568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018179.985238:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334413696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018179.985250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018179.985256:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018179/real 1734018179] req@ffff8800a934c700 x1818247334413696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018179.985266:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334413696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018179.985273:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018179.985275:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018179.985278:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334413696:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018180.704725:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018181.706792:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018182.708697:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018183.710691:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018184.712723:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018184.992601:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018184.992611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018184.992616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018184.992627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2451 00000100:00080000:0.0:1734018184.992636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018184.992644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018184.992681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334413824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018184.992715:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018184.992719:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018184.992721:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018184.992726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2451 00000100:00080000:0.0:1734018184.992729:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018184.992733:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018184.992743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018184.992754:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334413952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018184.992765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018184.992768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018184.992770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018184.992773:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2451 00000100:00080000:0.0:1734018184.992776:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018184.992780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018184.992784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018184.992793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334414080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018184.992801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018184.992804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018184.992805:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018184.992809:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2451 00000100:00080000:0.0:1734018184.992811:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018184.992815:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018184.992818:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018184.992826:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334414208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018184.992828:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334413824:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018184.992835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018184.992838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018184.992840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018184.992843:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2451 00000100:00080000:0.0:1734018184.992848:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018184.992851:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018184.992855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018184.992863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334414336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018184.992870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018184.992873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018184.992875:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018184.992878:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2451 00000100:00080000:0.0:1734018184.992881:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018184.992884:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018184.992888:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018184.992897:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334414464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018184.992905:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018184.992908:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018184.992909:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018184.992912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2451 00000100:00080000:0.0:1734018184.992915:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018184.992918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018184.992919:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334413952:192.168.201.116@tcp:38:kworker.0 00000008:00100000:0.0:1734018184.992922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018184.992929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334414592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018184.992940:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334414080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018184.992962:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334414208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018184.992977:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334414336:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018184.992991:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334414464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018184.993007:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334414592:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018185.714695:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018186.716714:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018187.718729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734018187.998799:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:2.0:1734018187.998859:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734018187.998884:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018187.998928:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018187.998936:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018187.998946:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018187.998954:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018187.998960:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018187.998967:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:0.0:1734018187.999037:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018184/real 1734018187] req@ffff8800a934d500 x1818247334414208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018187.999081:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334414208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018187.999101:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018187.999107:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018187.999115:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334414208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018187.999128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018184/real 1734018187] req@ffff8800a934fb80 x1818247334414336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018187.999148:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334414336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018187.999162:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018187.999166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018187.999172:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334414336:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018187.999184:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018184/real 1734018187] req@ffff8800a934f100 x1818247334414464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018187.999204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334414464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018187.999217:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018187.999221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018187.999226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334414464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018187.999238:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018184/real 1734018187] req@ffff8800a934e300 x1818247334414592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018187.999258:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334414592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018187.999271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018187.999274:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018187.999280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334414592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018187.999290:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018184/real 1734018187] req@ffff8800a934dc00 x1818247334413824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018187.999310:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334413824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018187.999360:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018187.999365:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018187.999371:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334413824:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018187.999383:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018184/real 1734018187] req@ffff8800a934d880 x1818247334413952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018187.999403:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334413952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018187.999415:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018187.999420:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018187.999425:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334413952:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018187.999435:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018184/real 1734018187] req@ffff8800a934df80 x1818247334414080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018187.999455:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334414080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018187.999527:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018187.999533:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018187.999539:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334414080:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018188.720728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018189.722753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018190.000613:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018190.000623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018190.000628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018190.000639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2456 00000100:00080000:0.0:1734018190.000648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018190.000656:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018190.000693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334414720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018190.000728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018190.000732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018190.000734:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018190.000739:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2456 00000100:00080000:0.0:1734018190.000742:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018190.000746:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018190.000756:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018190.000767:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334414848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018190.000783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018190.000786:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018190.000788:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018190.000792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2456 00000100:00080000:0.0:1734018190.000795:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018190.000798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018190.000803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018190.000811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334414976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018190.000819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018190.000823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018190.000824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018190.000827:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2456 00000100:00080000:0.0:1734018190.000830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018190.000833:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018190.000837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018190.000838:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334414720:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018190.000845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334415104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018190.000853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018190.000856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018190.000858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018190.000862:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2456 00000100:00080000:0.0:1734018190.000865:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018190.000868:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018190.000872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018190.000880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334415232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018190.000888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018190.000890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018190.000892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018190.000895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2456 00000100:00080000:0.0:1734018190.000898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018190.000901:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018190.000905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018190.000905:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018190.000913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334415360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018190.000921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018190.000924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018190.000925:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018190.000926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018190/real 1734018190] req@ffff8800a934df80 x1818247334414720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018190.000929:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2456 00000100:00080000:0.0:1734018190.000931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018190.000935:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018190.000938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018190.000946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334415488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018190.000946:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334414720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018190.000956:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018190.000960:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018190.000965:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334414720:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018190.000972:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334414848:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018190.000991:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018190.000997:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018190/real 1734018190] req@ffff8800a934d880 x1818247334414848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018190.001009:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334414848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018190.001017:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018190.001019:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018190.001022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334414848:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018190.001028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334414976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018190.001041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018190.001047:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018190/real 1734018190] req@ffff8800a934dc00 x1818247334414976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018190.001057:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334414976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018190.001065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018190.001067:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018190.001070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334414976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018190.001095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334415104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018190.001111:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018190.001117:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018190/real 1734018190] req@ffff8800a934e300 x1818247334415104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018190.001128:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334415104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018190.001136:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018190.001138:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018190.001141:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334415104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018190.001146:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334415232:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018190.001158:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018190.001164:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018190/real 1734018190] req@ffff8800a934f100 x1818247334415232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018190.001174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334415232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018190.001181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018190.001183:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018190.001186:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334415232:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018190.001191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334415360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018190.001203:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018190.001209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018190/real 1734018190] req@ffff8800a934fb80 x1818247334415360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018190.001225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334415360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018190.001233:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018190.001235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018190.001238:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334415360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018190.001243:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334415488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018190.001255:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018190.001261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018190/real 1734018190] req@ffff8800a934d500 x1818247334415488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018190.001271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334415488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018190.001278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018190.001280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018190.001283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334415488:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018190.724728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018191.726746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018192.728710:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018193.730706:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018194.732751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018195.008588:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018195.008597:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018195.008602:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018195.008613:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2461 00000100:00080000:0.0:1734018195.008622:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018195.008630:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018195.008667:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334415616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018195.008701:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018195.008705:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018195.008707:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018195.008712:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2461 00000100:00080000:0.0:1734018195.008715:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018195.008719:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018195.008729:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018195.008740:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334415744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018195.008750:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018195.008753:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018195.008755:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018195.008760:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2461 00000100:00080000:0.0:1734018195.008762:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018195.008766:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018195.008772:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018195.008782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334415872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018195.008791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018195.008794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018195.008796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018195.008799:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2461 00000100:00080000:0.0:1734018195.008801:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018195.008805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018195.008808:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018195.008808:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334415616:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018195.008817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334416000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018195.008826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018195.008830:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018195.008831:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018195.008834:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2461 00000100:00080000:0.0:1734018195.008837:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018195.008840:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018195.008844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018195.008851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334416128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018195.008859:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018195.008862:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018195.008864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018195.008867:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2461 00000100:00080000:0.0:1734018195.008870:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018195.008874:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734018195.008876:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018195.008878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018195.008885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334416256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018195.008893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018195.008896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018195.008897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018195.008898:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018195/real 1734018195] req@ffff8800a934ce00 x1818247334415616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018195.008901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2461 00000100:00080000:0.0:1734018195.008903:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018195.008906:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018195.008910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018195.008917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334416384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018195.008918:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334415616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018195.008929:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018195.008933:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018195.008938:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334415616:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018195.008945:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334415744:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018195.008962:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018195.008969:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018195/real 1734018195] req@ffff8800a934e680 x1818247334415744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018195.008979:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334415744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018195.008987:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018195.008991:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018195.008994:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334415744:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018195.009000:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334415872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018195.009013:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018195.009019:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018195/real 1734018195] req@ffff8800a934d500 x1818247334415872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018195.009029:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334415872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018195.009036:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018195.009039:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018195.009042:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334415872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018195.009067:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334416000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018195.009083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018195.009089:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018195/real 1734018195] req@ffff8800a934fb80 x1818247334416000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018195.009100:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334416000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018195.009107:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018195.009109:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018195.009112:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334416000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018195.009118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334416128:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018195.009134:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018195.009140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018195/real 1734018195] req@ffff8800a934f100 x1818247334416128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018195.009150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334416128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018195.009157:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018195.009159:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018195.009162:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334416128:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018195.009167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334416256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018195.009184:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018195.009191:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018195/real 1734018195] req@ffff8800a934e300 x1818247334416256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018195.009240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334416256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018195.009248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018195.009251:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018195.009254:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334416256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018195.009260:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334416384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018195.009276:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018195.009282:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018195/real 1734018195] req@ffff8800a934dc00 x1818247334416384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018195.009293:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334416384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018195.009300:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018195.009302:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018195.009305:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334416384:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018195.734723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018196.736755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018197.738734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018198.740730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018199.742715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018200.016618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018200.016627:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018200.016632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018200.016642:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2466 00000100:00080000:0.0:1734018200.016651:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018200.016658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018200.016695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334416512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018200.016736:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018200.016741:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018200.016743:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018200.016747:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2466 00000100:00080000:0.0:1734018200.016750:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018200.016754:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018200.016764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018200.016775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334416640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018200.016785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018200.016788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018200.016789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018200.016794:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2466 00000100:00080000:0.0:1734018200.016796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018200.016800:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018200.016804:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018200.016812:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334416768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018200.016821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018200.016824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018200.016825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018200.016829:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2466 00000100:00080000:0.0:1734018200.016831:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018200.016835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018200.016839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018200.016848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334416896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018200.016852:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334416512:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018200.016857:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018200.016860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018200.016861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018200.016864:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2466 00000100:00080000:0.0:1734018200.016867:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018200.016870:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018200.016874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018200.016881:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334417024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018200.016888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018200.016891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018200.016893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018200.016896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2466 00000100:00080000:0.0:1734018200.016899:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018200.016902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018200.016906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018200.016915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334417152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734018200.016920:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018200.016923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018200.016926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018200.016928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018200.016932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2466 00000100:00080000:0.0:1734018200.016935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018200.016938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018200.016942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018200.016944:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018200/real 1734018200] req@ffff8800a934d880 x1818247334416512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018200.016949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334417280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018200.016964:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334416512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018200.016975:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018200.016979:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018200.016984:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334416512:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018200.016991:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334416640:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018200.017007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018200.017014:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018200/real 1734018200] req@ffff8800a934df80 x1818247334416640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018200.017025:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334416640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018200.017033:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018200.017036:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018200.017039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334416640:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018200.017045:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334416768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018200.017058:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018200.017064:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018200/real 1734018200] req@ffff8800a934c700 x1818247334416768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018200.017074:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334416768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018200.017081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018200.017084:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018200.017087:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334416768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018200.017112:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334416896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018200.017192:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018200.017201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018200/real 1734018200] req@ffff8800a934c000 x1818247334416896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018200.017213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334416896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018200.017221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018200.017223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018200.017226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334416896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018200.017232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334417024:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018200.017244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018200.017250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018200/real 1734018200] req@ffff8800a934d180 x1818247334417024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018200.017261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334417024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018200.017269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018200.017271:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018200.017274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334417024:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018200.017279:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334417152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018200.017291:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018200.017297:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018200/real 1734018200] req@ffff8800a934ea00 x1818247334417152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018200.017313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334417152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018200.017321:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018200.017323:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018200.017326:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334417152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018200.017331:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334417280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018200.017343:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018200.017349:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018200/real 1734018200] req@ffff8800a934c380 x1818247334417280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018200.017358:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334417280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018200.017365:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018200.017367:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018200.017370:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334417280:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018200.744704:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018201.746755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018202.748694:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018203.750747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018204.752730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018205.024604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018205.024619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018205.024623:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018205.024638:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2471 00000100:00080000:0.0:1734018205.024647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018205.024654:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018205.024690:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334417408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018205.024724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018205.024728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018205.024730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018205.024735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2471 00000100:00080000:0.0:1734018205.024738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018205.024741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018205.024751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018205.024762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334417536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018205.024773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018205.024776:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018205.024778:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018205.024781:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2471 00000100:00080000:0.0:1734018205.024784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018205.024787:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018205.024792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018205.024802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334417664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018205.024811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018205.024814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018205.024815:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018205.024818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2471 00000100:00080000:0.0:1734018205.024821:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018205.024824:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018205.024828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018205.024831:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334417408:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018205.024836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334417792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018205.024846:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018205.024849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018205.024850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018205.024853:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2471 00000100:00080000:0.0:1734018205.024857:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018205.024860:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018205.024864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018205.024871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334417920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018205.024879:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018205.024882:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018205.024884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018205.024887:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2471 00000100:00080000:0.0:1734018205.024890:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018205.024893:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018205.024897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018205.024899:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018205.024904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334418048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018205.024912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018205.024915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018205.024917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018205.024920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2471 00000100:00100000:2.0:1734018205.024922:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018205/real 1734018205] req@ffff8800a934ed80 x1818247334417408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018205.024923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018205.024926:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018205.024930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018205.024937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334418176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018205.024941:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334417408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018205.024952:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018205.024956:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018205.024961:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334417408:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018205.024969:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334417536:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018205.024985:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018205.024993:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018205/real 1734018205] req@ffff8800a934ca80 x1818247334417536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018205.025004:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334417536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018205.025012:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018205.025014:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018205.025018:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334417536:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018205.025024:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334417664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018205.025074:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018205.025082:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018205/real 1734018205] req@ffff8800a934c380 x1818247334417664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018205.025093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334417664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018205.025100:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018205.025107:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018205.025110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334417664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018205.025139:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334417792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018205.025157:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018205.025163:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018205/real 1734018205] req@ffff8800a934ea00 x1818247334417792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018205.025174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334417792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018205.025182:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018205.025184:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018205.025187:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334417792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018205.025192:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334417920:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018205.025204:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018205.025210:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018205/real 1734018205] req@ffff8800a934d180 x1818247334417920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018205.025220:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334417920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018205.025227:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018205.025229:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018205.025232:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334417920:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018205.025238:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334418048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018205.025250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018205.025256:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018205/real 1734018205] req@ffff8800a934c000 x1818247334418048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018205.025273:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334418048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018205.025281:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018205.025283:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018205.025286:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334418048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018205.025291:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334418176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018205.025303:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018205.025309:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018205/real 1734018205] req@ffff8800a934c700 x1818247334418176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018205.025319:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334418176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018205.025326:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018205.025328:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018205.025331:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334418176:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018205.754765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018206.756753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018207.758731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018208.760750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018209.762716:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018210.032618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018210.032628:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018210.032633:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018210.032644:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2476 00000100:00080000:0.0:1734018210.032653:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018210.032661:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018210.032697:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334418304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018210.032731:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018210.032735:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018210.032737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018210.032742:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2476 00000100:00080000:0.0:1734018210.032745:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018210.032749:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018210.032758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018210.032770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334418432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018210.032779:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018210.032783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018210.032784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018210.032789:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2476 00000100:00080000:0.0:1734018210.032792:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018210.032795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018210.032801:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018210.032810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334418560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018210.032819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018210.032822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018210.032823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018210.032826:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2476 00000100:00080000:0.0:1734018210.032829:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018210.032832:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018210.032836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018210.032840:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334418304:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018210.032843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334418688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018210.032852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018210.032855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018210.032857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018210.032860:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2476 00000100:00080000:0.0:1734018210.032863:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018210.032866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018210.032869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018210.032877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334418816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018210.032885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018210.032888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018210.032890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018210.032893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2476 00000100:00080000:0.0:1734018210.032896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018210.032899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018210.032903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018210.032907:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018210.032913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334418944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018210.032921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018210.032924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018210.032925:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018210.032928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2476 00000100:00100000:2.0:1734018210.032929:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018210/real 1734018210] req@ffff8800a934df80 x1818247334418304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018210.032931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018210.032934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018210.032938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018210.032978:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334418304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018210.032983:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334419072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018210.032990:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018210.032994:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018210.032998:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334418304:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018210.033006:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334418432:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018210.033026:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018210.033035:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018210/real 1734018210] req@ffff8800a934d880 x1818247334418432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018210.033046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334418432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018210.033054:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018210.033057:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018210.033061:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334418432:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018210.033066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334418560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018210.033079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018210.033085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018210/real 1734018210] req@ffff8800a934dc00 x1818247334418560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018210.033095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334418560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018210.033103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018210.033105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018210.033109:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334418560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018210.033133:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334418688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018210.033149:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018210.033156:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018210/real 1734018210] req@ffff8800a934e300 x1818247334418688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018210.033167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334418688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018210.033174:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018210.033176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018210.033179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334418688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018210.033184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334418816:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018210.033196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018210.033202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018210/real 1734018210] req@ffff8800a934f100 x1818247334418816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018210.033212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334418816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018210.033219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018210.033221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018210.033224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334418816:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018210.033230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334418944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018210.033241:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018210.033247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018210/real 1734018210] req@ffff8800a934fb80 x1818247334418944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018210.033264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334418944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018210.033272:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018210.033274:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018210.033277:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334418944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018210.033282:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334419072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018210.033294:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018210.033300:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018210/real 1734018210] req@ffff8800a934d500 x1818247334419072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018210.033310:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334419072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018210.033317:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018210.033319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018210.033322:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334419072:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018210.764764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018211.766765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018212.768717:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018213.770689:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018214.772756:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018215.040606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018215.040616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018215.040620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018215.040632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2481 00000100:00080000:0.0:1734018215.040641:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018215.040649:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018215.040687:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334419200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018215.040728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018215.040733:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018215.040734:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018215.040739:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2481 00000100:00080000:0.0:1734018215.040742:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018215.040746:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018215.040756:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018215.040768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334419328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018215.040777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018215.040780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018215.040782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018215.040786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2481 00000100:00080000:0.0:1734018215.040789:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018215.040793:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018215.040797:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018215.040807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334419456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018215.040815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018215.040818:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018215.040820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018215.040823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2481 00000100:00080000:0.0:1734018215.040826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018215.040830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018215.040833:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334419200:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018215.040834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018215.040842:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334419584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018215.040851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018215.040854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018215.040855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018215.040859:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2481 00000100:00080000:0.0:1734018215.040898:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018215.040902:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018215.040906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018215.040916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334419712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018215.040924:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018215.040927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1734018215.040927:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018215.040929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018215.040932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2481 00000100:00080000:0.0:1734018215.040936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018215.040939:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018215.040943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018215.040951:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334419840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018215.040951:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018215/real 1734018215] req@ffff8800a934e680 x1818247334419200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018215.040959:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018215.040963:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018215.040965:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018215.040968:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2481 00000100:00080000:0.0:1734018215.040971:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018215.040971:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334419200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018215.040974:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018215.040978:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734018215.040982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1734018215.040985:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334419968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018215.040986:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018215.040991:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334419200:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018215.040999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334419328:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018215.041016:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018215.041023:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018215/real 1734018215] req@ffff8800a934ce00 x1818247334419328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018215.041035:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334419328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018215.041042:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018215.041045:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018215.041048:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334419328:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018215.041054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334419456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018215.041067:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018215.041073:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018215/real 1734018215] req@ffff8800a934d500 x1818247334419456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018215.041083:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334419456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018215.041090:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018215.041092:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018215.041095:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334419456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018215.041119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334419584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018215.041136:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018215.041142:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018215/real 1734018215] req@ffff8800a934fb80 x1818247334419584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018215.041153:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334419584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018215.041160:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018215.041162:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018215.041165:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334419584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018215.041170:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334419712:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018215.041182:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018215.041188:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018215/real 1734018215] req@ffff8800a934f100 x1818247334419712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018215.041198:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334419712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018215.041205:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018215.041207:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018215.041210:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334419712:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018215.041215:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334419840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018215.041227:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018215.041234:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018215/real 1734018215] req@ffff8800a934e300 x1818247334419840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018215.041250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334419840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018215.041258:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018215.041260:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018215.041263:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334419840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018215.041268:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334419968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018215.041280:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018215.041286:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018215/real 1734018215] req@ffff8800a934dc00 x1818247334419968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018215.041296:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334419968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018215.041302:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018215.041305:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018215.041308:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334419968:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018215.774754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018216.776752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018217.778723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018218.780862:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018219.782726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018220.048579:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018220.048589:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018220.048594:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018220.048605:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2486 00000100:00080000:0.0:1734018220.048614:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018220.048622:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018220.048659:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334420096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018220.048693:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018220.048697:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018220.048698:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018220.048703:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2486 00000100:00080000:0.0:1734018220.048706:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018220.048710:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018220.048720:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018220.048731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334420224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018220.048742:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018220.048745:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018220.048747:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018220.048751:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2486 00000100:00080000:0.0:1734018220.048753:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018220.048757:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018220.048762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018220.048770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334420352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018220.048818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018220.048821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018220.048823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018220.048826:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2486 00000100:00080000:0.0:1734018220.048829:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018220.048833:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018220.048837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018220.048837:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334420096:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018220.048846:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334420480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018220.048856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018220.048859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018220.048860:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018220.048864:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2486 00000100:00080000:0.0:1734018220.048868:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018220.048871:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018220.048875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018220.048882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334420608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018220.048891:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018220.048893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018220.048895:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018220.048898:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2486 00000100:00080000:0.0:1734018220.048901:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734018220.048903:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018220.048904:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018220.048908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018220.048917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334420736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018220.048924:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734018220.048926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018220/real 1734018220] req@ffff8800a934d880 x1818247334420096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018220.048928:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018220.048929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018220.048932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2486 00000100:00080000:0.0:1734018220.048935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018220.048938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018220.048942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018220.048946:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334420096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018220.048950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334420864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018220.048956:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018220.048960:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018220.048965:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334420096:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018220.048973:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334420224:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018220.048990:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018220.048996:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018220/real 1734018220] req@ffff8800a934df80 x1818247334420224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018220.049007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334420224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018220.049015:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018220.049018:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018220.049022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334420224:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018220.049027:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334420352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018220.049041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018220.049047:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018220/real 1734018220] req@ffff8800a934c700 x1818247334420352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018220.049057:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334420352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018220.049064:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018220.049066:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018220.049070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334420352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018220.049094:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334420480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018220.049110:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018220.049117:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018220/real 1734018220] req@ffff8800a934c000 x1818247334420480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018220.049127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334420480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018220.049135:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018220.049137:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018220.049140:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334420480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018220.049145:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334420608:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018220.049157:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018220.049163:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018220/real 1734018220] req@ffff8800a934d180 x1818247334420608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018220.049173:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334420608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018220.049180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018220.049182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018220.049185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334420608:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018220.049191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334420736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018220.049203:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018220.049209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018220/real 1734018220] req@ffff8800a934ea00 x1818247334420736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018220.049225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334420736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018220.049233:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018220.049235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018220.049238:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334420736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018220.049243:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334420864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018220.049255:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018220.049261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018220/real 1734018220] req@ffff8800a934c380 x1818247334420864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018220.049270:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334420864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018220.049277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018220.049279:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018220.049282:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334420864:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018220.784745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018221.786735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018222.788704:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018223.790803:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018224.792759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018225.056597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018225.056607:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018225.056612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018225.056622:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2491 00000100:00080000:0.0:1734018225.056631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018225.056639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018225.056675:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334420992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018225.056759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018225.056766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018225.056770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018225.056777:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2491 00000100:00080000:0.0:1734018225.056782:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018225.056789:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018225.056802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018225.056823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334421120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018225.056830:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334420992:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018225.056840:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018225.056844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018225.056846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018225.056852:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2491 00000100:00080000:0.0:1734018225.056856:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018225.056861:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018225.056869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018225.056883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334421248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018225.056896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1734018225.056900:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018225.056901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018225.056904:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018225.056909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2491 00000100:00080000:0.0:1734018225.056913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018225.056917:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018225.056923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018225.056924:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018225/real 1734018225] req@ffff8800a934ca80 x1818247334420992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018225.056935:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334421376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018225.056948:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734018225.056948:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334420992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018225.056954:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018225.056956:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018225.056961:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2491 00000100:00080000:2.0:1734018225.056963:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018225.056965:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018225.056968:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734018225.056970:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018225.056974:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334420992:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018225.056976:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018225.056987:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334421504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018225.056999:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734018225.057001:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334421120:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734018225.057004:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018225.057007:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018225.057013:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2491 00000100:00080000:0.0:1734018225.057017:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018225.057022:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734018225.057026:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018225.057028:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018225.057037:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018225/real 1734018225] req@ffff8800a934ed80 x1818247334421120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018225.057040:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334421632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018225.057054:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734018225.057054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334421120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018225.057059:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018225.057062:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018225.057067:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018225.057068:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2491 00000100:00080000:2.0:1734018225.057071:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734018225.057072:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018225.057077:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018225.057077:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334421120:192.168.201.116@tcp:38:kworker.0 00000008:00100000:0.0:1734018225.057083:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018225.057086:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334421248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018225.057095:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334421760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734018225.057107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018225.057117:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018225/real 1734018225] req@ffff8800a934c380 x1818247334421248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018225.057134:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334421248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018225.057146:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018225.057150:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018225.057155:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334421248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018225.057165:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334421376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018225.057194:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018225.057207:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018225/real 1734018225] req@ffff8800a934ea00 x1818247334421376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018225.057224:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334421376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018225.057237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018225.057240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018225.057245:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334421376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018225.057265:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334421504:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018225.057284:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018225.057292:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018225/real 1734018225] req@ffff8800a934d180 x1818247334421504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018225.057303:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334421504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018225.057310:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018225.057312:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018225.057316:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334421504:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018225.057321:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334421632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018225.057333:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018225.057339:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018225/real 1734018225] req@ffff8800a934c000 x1818247334421632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018225.057356:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334421632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018225.057364:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018225.057366:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018225.057369:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334421632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018225.057374:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334421760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018225.057386:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018225.057392:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018225/real 1734018225] req@ffff8800a934c700 x1818247334421760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018225.057402:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334421760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018225.057409:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018225.057411:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018225.057414:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334421760:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018225.794749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018226.796708:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018227.798759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018228.800724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018229.802783:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018230.064682:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018230.064690:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018230.064695:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018230.064705:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2496 00000100:00080000:3.0:1734018230.064713:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018230.064721:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018230.064758:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334421888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018230.064794:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018230.064798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018230.064800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018230.064804:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2496 00000100:00080000:3.0:1734018230.064808:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018230.064811:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018230.064821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018230.064833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334422016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018230.064843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018230.064846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018230.064847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018230.064852:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2496 00000100:00080000:3.0:1734018230.064854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018230.064858:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018230.064863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018230.064871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334422144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018230.064880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018230.064883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018230.064885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018230.064889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2496 00000100:00080000:3.0:1734018230.064892:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018230.064894:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334421888:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018230.064895:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018230.064899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018230.064906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334422272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018230.064914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018230.064917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018230.064919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018230.064922:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2496 00000100:00080000:3.0:1734018230.064925:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018230.064928:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018230.064932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018230.064939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334422400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018230.064947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018230.064950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018230.064951:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018230.064954:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2496 00000100:00080000:3.0:1734018230.064957:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018230.064960:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734018230.064961:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:3.0:1734018230.064964:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018230.064974:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334422528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018230.064982:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734018230.064983:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018230/real 1734018230] req@ffff8800a7983800 x1818247334421888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018230.064986:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018230.064987:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018230.064991:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2496 00000100:00080000:3.0:1734018230.064993:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018230.064996:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018230.065000:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018230.065003:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334421888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018230.065007:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334422656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018230.065014:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018230.065017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018230.065022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334421888:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018230.065029:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334422016:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018230.065046:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018230.065052:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018230/real 1734018230] req@ffff8800a7980e00 x1818247334422016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018230.065064:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334422016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018230.065072:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018230.065074:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018230.065077:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334422016:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018230.065100:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334422144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018230.065113:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018230.065119:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018230/real 1734018230] req@ffff8800a7983b80 x1818247334422144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018230.065129:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334422144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018230.065137:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018230.065139:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018230.065143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334422144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018230.065149:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334422272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018230.065164:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018230.065170:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018230/real 1734018230] req@ffff8800a7980700 x1818247334422272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018230.065180:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334422272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018230.065187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018230.065189:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018230.065192:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334422272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018230.065197:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334422400:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018230.065214:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018230.065220:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018230/real 1734018230] req@ffff8800a7981180 x1818247334422400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018230.065230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334422400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018230.065237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018230.065239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018230.065242:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334422400:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018230.065247:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334422528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018230.065264:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018230.065271:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018230/real 1734018230] req@ffff8800a7982d80 x1818247334422528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018230.065287:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334422528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018230.065295:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018230.065297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018230.065300:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334422528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018230.065305:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334422656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018230.065316:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018230.065322:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018230/real 1734018230] req@ffff8800a7981500 x1818247334422656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018230.065332:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334422656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018230.065339:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018230.065341:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018230.065344:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334422656:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018230.804748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018231.806697:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018232.808766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018233.810739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018234.812785:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018235.072631:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018235.072640:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018235.072644:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018235.072655:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2501 00000100:00080000:3.0:1734018235.072664:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018235.072671:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018235.072707:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334422784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018235.072740:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018235.072744:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018235.072746:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018235.072750:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2501 00000100:00080000:3.0:1734018235.072754:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018235.072757:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018235.072767:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018235.072779:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334422912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018235.072789:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018235.072792:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018235.072794:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018235.072798:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2501 00000100:00080000:3.0:1734018235.072801:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018235.072804:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018235.072809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018235.072817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334423040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018235.072826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018235.072829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018235.072831:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018235.072834:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2501 00000100:00080000:3.0:1734018235.072836:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018235.072840:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018235.072844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018235.072847:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334422784:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018235.072853:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334423168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018235.072862:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018235.072865:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018235.072867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018235.072870:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2501 00000100:00080000:3.0:1734018235.072873:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018235.072876:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018235.072880:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018235.072887:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334423296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018235.072895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018235.072898:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018235.072899:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018235.072903:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2501 00000100:00080000:3.0:1734018235.072905:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018235.072909:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018235.072913:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018235.072914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734018235.072920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334423424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018235.072928:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018235.072931:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018235.072932:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018235.072936:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018235/real 1734018235] req@ffff8800a7982680 x1818247334422784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018235.072936:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2501 00000100:00080000:3.0:1734018235.072939:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018235.072942:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018235.072946:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018235.072953:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334423552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018235.072956:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334422784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018235.072967:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018235.072970:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018235.072975:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334422784:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018235.072983:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334422912:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018235.072999:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018235.073006:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018235/real 1734018235] req@ffff8800a7981880 x1818247334422912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018235.073017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334422912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018235.073025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018235.073027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018235.073031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334422912:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018235.073037:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334423040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018235.073050:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018235.073056:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018235/real 1734018235] req@ffff8800a7980a80 x1818247334423040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018235.073066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334423040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018235.073073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018235.073076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018235.073079:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334423040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018235.073103:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334423168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018235.073119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018235.073125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018235/real 1734018235] req@ffff8800a7981f80 x1818247334423168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018235.073136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334423168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018235.073143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018235.073146:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018235.073149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334423168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018235.073154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334423296:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018235.073166:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018235.073172:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018235/real 1734018235] req@ffff8800a7982300 x1818247334423296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018235.073182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334423296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018235.073189:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018235.073191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018235.073194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334423296:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018235.073199:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334423424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018235.073211:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018235.073217:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018235/real 1734018235] req@ffff8800a7980000 x1818247334423424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018235.073234:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334423424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018235.073241:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018235.073243:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018235.073246:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334423424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018235.073251:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334423552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018235.073263:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018235.073269:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018235/real 1734018235] req@ffff8800a7983480 x1818247334423552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018235.073279:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334423552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018235.073286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018235.073288:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018235.073291:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334423552:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018235.814732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018236.816741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018237.818749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018238.820755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018239.822716:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018240.080598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018240.080607:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018240.080611:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018240.080621:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2506 00000100:00080000:3.0:1734018240.080630:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018240.080637:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018240.080674:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334423680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018240.080707:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018240.080711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018240.080713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018240.080717:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2506 00000100:00080000:3.0:1734018240.080721:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018240.080724:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018240.080735:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018240.080747:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334423808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018240.080758:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018240.080761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018240.080763:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018240.080767:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2506 00000100:00080000:3.0:1734018240.080770:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018240.080774:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018240.080778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018240.080789:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334423936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018240.080797:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018240.080800:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018240.080802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018240.080805:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2506 00000100:00080000:3.0:1734018240.080808:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018240.080811:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018240.080813:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334423680:192.168.201.116@tcp:250:kworker.0 00000008:00100000:3.0:1734018240.080815:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018240.080824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334424064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018240.080834:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018240.080836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018240.080838:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018240.080843:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2506 00000100:00080000:3.0:1734018240.080845:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018240.080848:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018240.080852:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018240.080860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334424192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018240.080868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018240.080871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018240.080873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018240.080876:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2506 00000800:00000100:2.0:1734018240.080878:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018240.080879:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018240.080882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018240.080886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018240.080894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334424320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018240.080899:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018240/real 1734018240] req@ffff8800a7981c00 x1818247334423680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018240.080901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018240.080905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018240.080906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018240.080909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2506 00000100:00080000:3.0:1734018240.080912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018240.080915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018240.080919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018240.080920:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334423680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018240.080927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334424448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018240.080931:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018240.080935:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018240.080940:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334423680:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018240.080947:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334423808:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018240.080964:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018240.080971:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018240/real 1734018240] req@ffff8800a7980380 x1818247334423808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018240.080982:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334423808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018240.080990:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018240.080993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018240.080996:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334423808:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018240.081002:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334423936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018240.081015:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018240.081021:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018240/real 1734018240] req@ffff8800a7983480 x1818247334423936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018240.081031:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334423936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018240.081038:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018240.081040:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018240.081043:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334423936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018240.081070:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334424064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018240.081087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018240.081093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018240/real 1734018240] req@ffff8800a7980000 x1818247334424064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018240.081104:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334424064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018240.081112:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018240.081114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018240.081117:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334424064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018240.081122:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334424192:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018240.081135:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018240.081141:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018240/real 1734018240] req@ffff8800a7982300 x1818247334424192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018240.081151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334424192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018240.081158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018240.081160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018240.081163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334424192:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018240.081168:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334424320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018240.081180:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018240.081186:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018240/real 1734018240] req@ffff8800a7981f80 x1818247334424320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018240.081202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334424320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018240.081209:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018240.081211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018240.081214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334424320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018240.081219:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334424448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018240.081231:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018240.081237:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018240/real 1734018240] req@ffff8800a7980a80 x1818247334424448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018240.081247:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334424448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018240.081254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018240.081256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018240.081259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334424448:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018240.824739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018241.826768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018242.828744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018243.830738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018244.832731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018245.088609:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018245.088617:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018245.088622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018245.088633:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2511 00000100:00080000:3.0:1734018245.088642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018245.088649:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018245.088686:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334424576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018245.088720:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018245.088724:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018245.088726:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018245.088730:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2511 00000100:00080000:3.0:1734018245.088733:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018245.088737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018245.088747:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018245.088759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334424704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018245.088768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018245.088771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018245.088773:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018245.088778:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2511 00000100:00080000:3.0:1734018245.088780:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018245.088784:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018245.088789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018245.088797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334424832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018245.088805:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018245.088808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018245.088810:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018245.088813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2511 00000100:00080000:3.0:1734018245.088815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018245.088819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018245.088823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018245.088824:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334424576:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018245.088832:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334424960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018245.088841:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018245.088844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018245.088845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018245.088848:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2511 00000100:00080000:3.0:1734018245.088851:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018245.088854:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018245.088858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018245.088866:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334425088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018245.088874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018245.088877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018245.088879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018245.088882:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2511 00000100:00080000:3.0:1734018245.088885:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018245.088889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734018245.088890:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:3.0:1734018245.088893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018245.088900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334425216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018245.088907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018245.088910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018245.088912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018245.088913:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018245/real 1734018245] req@ffff8800a7981880 x1818247334424576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018245.088915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2511 00000100:00080000:3.0:1734018245.088918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018245.088921:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018245.088925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018245.088932:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334425344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018245.088933:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334424576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018245.088943:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018245.088947:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018245.088952:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334424576:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018245.088959:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334424704:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018245.088976:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018245.088982:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018245/real 1734018245] req@ffff8800a7982680 x1818247334424704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018245.088994:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334424704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018245.089002:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018245.089004:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018245.089008:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334424704:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018245.089014:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334424832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018245.089027:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018245.089032:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018245/real 1734018245] req@ffff8800a7981500 x1818247334424832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018245.089042:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334424832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018245.089050:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018245.089052:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018245.089055:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334424832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018245.089080:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334424960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018245.089096:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018245.089102:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018245/real 1734018245] req@ffff8800a7982d80 x1818247334424960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018245.089113:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334424960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018245.089120:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018245.089122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018245.089125:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334424960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018245.089130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334425088:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018245.089143:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018245.089149:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018245/real 1734018245] req@ffff8800a7981180 x1818247334425088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018245.089158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334425088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018245.089166:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018245.089168:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018245.089171:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334425088:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018245.089176:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334425216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018245.089193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018245.089199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018245/real 1734018245] req@ffff8800a7980700 x1818247334425216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018245.089215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334425216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018245.089223:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018245.089225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018245.089228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334425216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018245.089233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334425344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018245.089245:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018245.089251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018245/real 1734018245] req@ffff8800a7983b80 x1818247334425344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018245.089260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334425344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018245.089267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018245.089270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018245.089273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334425344:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018245.834785:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018246.836789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018247.838744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018248.840704:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018249.842776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018250.096603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018250.096613:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018250.096617:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018250.096628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2516 00000100:00080000:3.0:1734018250.096636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018250.096644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018250.096681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334425472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018250.096715:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018250.096719:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018250.096721:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018250.096726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2516 00000100:00080000:3.0:1734018250.096729:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018250.096733:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018250.096743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018250.096755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334425600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018250.096764:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018250.096768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018250.096770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018250.096774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2516 00000100:00080000:3.0:1734018250.096776:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018250.096780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018250.096784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018250.096794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334425728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018250.096804:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018250.096808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018250.096809:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018250.096813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2516 00000100:00080000:3.0:1734018250.096815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018250.096818:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334425472:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018250.096818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018250.096823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018250.096831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334425856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018250.096839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018250.096842:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018250.096843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018250.096847:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2516 00000100:00080000:3.0:1734018250.096849:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018250.096852:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018250.096856:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018250.096863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334425984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018250.096871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018250.096874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018250.096875:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018250.096879:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2516 00000100:00080000:3.0:1734018250.096883:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018250.096886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018250.096890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018250.096898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334426112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018250.096906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734018250.096908:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334425600:192.168.201.116@tcp:38:kworker.0 00000100:00080000:3.0:1734018250.096909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018250.096910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018250.096914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2516 00000100:00080000:3.0:1734018250.096916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018250.096919:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018250.096923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018250.096930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334426240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018250.096936:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334425728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018250.096951:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334425856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018250.096966:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334425984:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018250.096982:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334426112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018250.096998:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334426240:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018250.844758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018251.846749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018252.848789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734018253.102939:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:2.0:1734018253.103012:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734018253.103036:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018253.103084:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018253.103094:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018253.103102:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018253.103110:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018253.103118:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018253.103125:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:3.0:1734018253.103178:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018250/real 1734018253] req@ffff8800a7981500 x1818247334426240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018253.103260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334426240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018253.103278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734018253.103284:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734018253.103292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334426240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734018253.103306:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018250/real 1734018253] req@ffff8800a7983b80 x1818247334425728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018253.103329:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334425728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018253.103343:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734018253.103347:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734018253.103354:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334425728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734018253.103366:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018250/real 1734018253] req@ffff8800a7980700 x1818247334425856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018253.103385:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334425856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018253.103396:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734018253.103399:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734018253.103404:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334425856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734018253.103414:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018250/real 1734018253] req@ffff8800a7981180 x1818247334425984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018253.103433:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334425984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018253.103445:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734018253.103449:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734018253.103454:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334425984:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734018253.103526:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018250/real 1734018253] req@ffff8800a7982d80 x1818247334426112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018253.103549:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334426112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018253.103561:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734018253.103565:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734018253.103570:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334426112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:3.0:1734018253.103580:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018250/real 1734018253] req@ffff8800a7980e00 x1818247334425472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018253.103597:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334425472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018253.103608:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734018253.103612:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734018253.103617:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334425472:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018253.103628:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018250/real 1734018253] req@ffff8800a7983800 x1818247334425600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018253.103644:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334425600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018253.103656:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734018253.103660:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734018253.103666:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334425600:192.168.201.116@tcp:38:kworker.0 00010000:00010000:3.0:1734018253.850722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018254.852750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018255.104604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018255.104612:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018255.104617:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018255.104628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2521 00000100:00080000:3.0:1734018255.104637:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018255.104644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018255.104682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334426368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018255.104717:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018255.104721:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018255.104723:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018255.104727:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2521 00000100:00080000:3.0:1734018255.104731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018255.104735:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018255.104745:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018255.104757:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334426496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018255.104766:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018255.104769:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018255.104771:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018255.104776:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2521 00000100:00080000:3.0:1734018255.104778:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018255.104782:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018255.104787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018255.104796:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334426624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018255.104806:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018255.104809:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018255.104810:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018255.104814:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2521 00000100:00080000:3.0:1734018255.104816:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734018255.104817:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334426368:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018255.104819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018255.104823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018255.104831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334426752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018255.104839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018255.104842:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018255.104843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018255.104846:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2521 00000100:00080000:3.0:1734018255.104848:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018255.104852:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018255.104856:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018255.104863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334426880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018255.104872:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018255.104874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018255.104876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018255.104880:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2521 00000800:00000100:0.0:1734018255.104883:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018255.104883:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018255.104886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018255.104890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018255.104897:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334427008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018255.104905:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018255.104908:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734018255.104909:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018255/real 1734018255] req@ffff8800a7983800 x1818247334426368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018255.104909:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018255.104912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2521 00000100:00080000:3.0:1734018255.104915:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018255.104918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018255.104922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018255.104929:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334426368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018255.104929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334427136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018255.104939:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018255.104943:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018255.104948:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334426368:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018255.104955:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334426496:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018255.104972:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018255.104979:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018255/real 1734018255] req@ffff8800a7980e00 x1818247334426496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018255.104991:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334426496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018255.104998:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018255.105001:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018255.105005:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334426496:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018255.105027:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334426624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018255.105041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018255.105048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018255/real 1734018255] req@ffff8800a7982d80 x1818247334426624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018255.105058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334426624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018255.105066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018255.105068:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018255.105071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334426624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018255.105077:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334426752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018255.105092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018255.105099:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018255/real 1734018255] req@ffff8800a7981180 x1818247334426752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018255.105109:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334426752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018255.105117:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018255.105119:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018255.105122:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334426752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018255.105127:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334426880:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018255.105139:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018255.105145:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018255/real 1734018255] req@ffff8800a7980700 x1818247334426880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018255.105155:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334426880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018255.105162:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018255.105164:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018255.105167:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334426880:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018255.105173:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334427008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018255.105221:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018255.105229:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018255/real 1734018255] req@ffff8800a7983b80 x1818247334427008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018255.105247:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334427008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018255.105255:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018255.105257:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018255.105260:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334427008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018255.105266:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334427136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018255.105279:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018255.105285:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018255/real 1734018255] req@ffff8800a7981500 x1818247334427136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018255.105295:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334427136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018255.105302:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018255.105305:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018255.105308:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334427136:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018255.854713:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018256.856738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018257.858714:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018258.860730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018259.862761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018260.112617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018260.112626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018260.112630:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018260.112641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2526 00000100:00080000:3.0:1734018260.112651:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018260.112658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018260.112695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334427264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018260.112730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018260.112734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018260.112736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018260.112741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2526 00000100:00080000:3.0:1734018260.112744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018260.112748:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018260.112758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018260.112770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334427392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018260.112780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018260.112783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018260.112785:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018260.112789:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2526 00000100:00080000:3.0:1734018260.112791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018260.112795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018260.112800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018260.112808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334427520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018260.112817:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018260.112823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018260.112825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018260.112828:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2526 00000100:00100000:0.0:1734018260.112831:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334427264:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018260.112831:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018260.112834:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018260.112838:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018260.112848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334427648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018260.112856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018260.112859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018260.112861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018260.112864:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2526 00000100:00080000:3.0:1734018260.112866:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018260.112869:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018260.112873:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018260.112881:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334427776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018260.112889:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018260.112892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018260.112893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734018260.112897:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018260.112897:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2526 00000100:00080000:3.0:1734018260.112900:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018260.112903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018260.112908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018260.112915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334427904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018260.112923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734018260.112924:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018260/real 1734018260] req@ffff8800a7982680 x1818247334427264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018260.112926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018260.112928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018260.112931:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2526 00000100:00080000:3.0:1734018260.112934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018260.112937:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018260.112941:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018260.112944:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334427264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018260.112948:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334428032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018260.112955:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018260.112959:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018260.112964:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334427264:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018260.112972:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334427392:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018260.112989:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018260.112996:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018260/real 1734018260] req@ffff8800a7981880 x1818247334427392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018260.113007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334427392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018260.113015:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018260.113018:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018260.113022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334427392:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018260.113045:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334427520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018260.113059:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018260.113065:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018260/real 1734018260] req@ffff8800a7980a80 x1818247334427520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018260.113076:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334427520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018260.113084:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018260.113086:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018260.113089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334427520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018260.113125:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334427648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018260.113146:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018260.113154:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018260/real 1734018260] req@ffff8800a7981f80 x1818247334427648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018260.113165:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334427648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018260.113173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018260.113175:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018260.113178:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334427648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018260.113183:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334427776:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018260.113196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018260.113203:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018260/real 1734018260] req@ffff8800a7982300 x1818247334427776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018260.113212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334427776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018260.113220:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018260.113222:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018260.113225:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334427776:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018260.113230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334427904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018260.113242:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018260.113249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018260/real 1734018260] req@ffff8800a7980000 x1818247334427904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018260.113265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334427904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018260.113273:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018260.113275:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018260.113278:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334427904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018260.113283:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334428032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018260.113295:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018260.113301:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018260/real 1734018260] req@ffff8800a7983480 x1818247334428032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018260.113311:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334428032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018260.113318:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018260.113320:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018260.113323:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334428032:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018260.864739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018261.866702:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018262.868672:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018263.870742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018264.872712:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018265.120608:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018265.120617:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018265.120622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018265.120632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2531 00000100:00080000:3.0:1734018265.120641:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018265.120648:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018265.120684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334428160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018265.120725:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018265.120729:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018265.120731:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018265.120735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2531 00000100:00080000:3.0:1734018265.120739:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018265.120743:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018265.120753:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018265.120765:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334428288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018265.120774:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018265.120777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018265.120778:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018265.120782:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2531 00000100:00080000:3.0:1734018265.120785:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018265.120789:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018265.120794:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018265.120804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334428416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018265.120813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018265.120816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018265.120817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018265.120821:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2531 00000100:00080000:3.0:1734018265.120824:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734018265.120825:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334428160:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018265.120827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018265.120832:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018265.120840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334428544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018265.120848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018265.120851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018265.120853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018265.120856:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2531 00000100:00080000:3.0:1734018265.120858:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018265.120861:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018265.120865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018265.120873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334428672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018265.120880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018265.120883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018265.120885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018265.120888:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2531 00000800:00000100:0.0:1734018265.120891:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018265.120891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018265.120894:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018265.120898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018265.120905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334428800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018265.120913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018265.120917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734018265.120918:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018265/real 1734018265] req@ffff8800a7980380 x1818247334428160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018265.120918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018265.120922:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2531 00000100:00080000:3.0:1734018265.120925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018265.120928:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018265.120932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018265.120939:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334428160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018265.120939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334428928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018265.120949:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018265.120953:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018265.120958:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334428160:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018265.120965:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334428288:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018265.120982:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018265.120989:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018265/real 1734018265] req@ffff8800a7981c00 x1818247334428288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018265.121001:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334428288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018265.121049:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018265.121052:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018265.121056:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334428288:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018265.121081:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334428416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018265.121099:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018265.121108:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018265/real 1734018265] req@ffff8800a7983480 x1818247334428416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018265.121119:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334428416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018265.121126:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018265.121129:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018265.121132:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334428416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018265.121137:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334428544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018265.121153:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018265.121159:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018265/real 1734018265] req@ffff8800a7980000 x1818247334428544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018265.121170:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334428544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018265.121178:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018265.121180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018265.121183:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334428544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018265.121188:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334428672:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018265.121201:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018265.121207:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018265/real 1734018265] req@ffff8800a7982300 x1818247334428672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018265.121217:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334428672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018265.121224:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018265.121226:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018265.121229:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334428672:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018265.121234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334428800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018265.121247:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018265.121253:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018265/real 1734018265] req@ffff8800a7981f80 x1818247334428800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018265.121270:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334428800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018265.121278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018265.121280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018265.121283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334428800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018265.121288:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334428928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018265.121300:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018265.121306:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018265/real 1734018265] req@ffff8800a7980a80 x1818247334428928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018265.121316:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334428928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018265.121323:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018265.121325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018265.121328:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334428928:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018265.874750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018266.876762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018267.878730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018268.880731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018269.882773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018270.128614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018270.128623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018270.128628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018270.128639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2536 00000100:00080000:3.0:1734018270.128647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018270.128655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018270.128692:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334429056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018270.128726:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018270.128730:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018270.128732:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018270.128737:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2536 00000100:00080000:3.0:1734018270.128740:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018270.128744:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018270.128755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018270.128766:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334429184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018270.128776:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018270.128780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018270.128781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018270.128786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2536 00000100:00080000:3.0:1734018270.128788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018270.128792:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018270.128796:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018270.128805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334429312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018270.128813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018270.128816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018270.128818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018270.128821:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2536 00000100:00080000:3.0:1734018270.128824:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018270.128827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018270.128831:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018270.128833:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334429056:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018270.128840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334429440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018270.128856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018270.128859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018270.128860:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018270.128863:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2536 00000100:00080000:3.0:1734018270.128866:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018270.128869:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018270.128873:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018270.128880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334429568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018270.128888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018270.128891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018270.128892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018270.128896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2536 00000800:00000100:0.0:1734018270.128898:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018270.128898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018270.128902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018270.128905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018270.128913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334429696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018270.128921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734018270.128950:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018270/real 1734018270] req@ffff8800a7981880 x1818247334429056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018270.128950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018270.128952:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018270.128956:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2536 00000100:00080000:3.0:1734018270.128960:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018270.128963:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018270.128967:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018270.128972:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334429056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018270.128976:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334429824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018270.128983:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018270.128987:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018270.128992:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334429056:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018270.129000:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334429184:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018270.129020:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018270.129028:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018270/real 1734018270] req@ffff8800a7982680 x1818247334429184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018270.129039:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334429184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018270.129047:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018270.129050:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018270.129053:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334429184:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018270.129059:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334429312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018270.129072:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018270.129079:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018270/real 1734018270] req@ffff8800a7981500 x1818247334429312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018270.129089:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334429312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018270.129096:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018270.129098:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018270.129101:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334429312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018270.129130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334429440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018270.129147:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018270.129154:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018270/real 1734018270] req@ffff8800a7983b80 x1818247334429440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018270.129164:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334429440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018270.129171:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018270.129174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018270.129176:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334429440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018270.129182:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334429568:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018270.129194:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018270.129200:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018270/real 1734018270] req@ffff8800a7980700 x1818247334429568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018270.129210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334429568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018270.129217:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018270.129219:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018270.129222:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334429568:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018270.129228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334429696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018270.129240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018270.129246:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018270/real 1734018270] req@ffff8800a7981180 x1818247334429696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018270.129263:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334429696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018270.129270:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018270.129272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018270.129275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334429696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018270.129280:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334429824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018270.129292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018270.129298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018270/real 1734018270] req@ffff8800a7982d80 x1818247334429824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018270.129308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334429824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018270.129315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018270.129317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018270.129320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334429824:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018270.884713:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018271.886750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018272.888750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018273.890710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018274.892730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018275.136573:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018275.136582:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018275.136587:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018275.136597:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2541 00000100:00080000:3.0:1734018275.136605:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018275.136613:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018275.136648:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334429952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018275.136682:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018275.136686:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018275.136688:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018275.136692:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2541 00000100:00080000:3.0:1734018275.136696:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018275.136699:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018275.136710:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018275.136721:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334430080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018275.136731:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018275.136734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018275.136736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018275.136740:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2541 00000100:00080000:3.0:1734018275.136742:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018275.136746:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018275.136751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018275.136761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334430208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018275.136769:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018275.136772:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018275.136774:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018275.136777:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2541 00000100:00080000:3.0:1734018275.136780:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018275.136783:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018275.136787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018275.136789:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334429952:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018275.136795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334430336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018275.136804:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018275.136807:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018275.136808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018275.136813:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2541 00000100:00080000:3.0:1734018275.136816:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018275.136819:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018275.136823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018275.136830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334430464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018275.136880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018275.136884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018275.136886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734018275.136887:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018275.136890:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2541 00000100:00080000:3.0:1734018275.136893:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018275.136896:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018275.136901:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018275.136910:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334430592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018275.136915:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018275/real 1734018275] req@ffff8800a7980e00 x1818247334429952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018275.136919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018275.136922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018275.136924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018275.136927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2541 00000100:00080000:3.0:1734018275.136930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018275.136933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734018275.136936:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334429952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018275.136937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018275.136944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334430720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018275.136947:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018275.136951:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018275.136956:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334429952:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018275.136964:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334430080:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018275.136983:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018275.136991:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018275/real 1734018275] req@ffff8800a7983800 x1818247334430080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018275.137002:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334430080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018275.137010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018275.137012:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018275.137015:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334430080:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018275.137021:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334430208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018275.137034:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018275.137041:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018275/real 1734018275] req@ffff8800a7982d80 x1818247334430208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018275.137051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334430208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018275.137058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018275.137060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018275.137063:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334430208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018275.137088:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334430336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018275.137105:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018275.137112:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018275/real 1734018275] req@ffff8800a7981180 x1818247334430336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018275.137122:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334430336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018275.137129:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018275.137132:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018275.137135:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334430336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018275.137140:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334430464:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018275.137152:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018275.137158:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018275/real 1734018275] req@ffff8800a7980700 x1818247334430464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018275.137168:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334430464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018275.137175:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018275.137178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018275.137181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334430464:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018275.137186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334430592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018275.137198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018275.137205:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018275/real 1734018275] req@ffff8800a7983b80 x1818247334430592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018275.137221:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334430592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018275.137229:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018275.137231:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018275.137234:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334430592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018275.137239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334430720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018275.137251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018275.137257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018275/real 1734018275] req@ffff8800a7981500 x1818247334430720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018275.137267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334430720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018275.137274:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018275.137276:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018275.137279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334430720:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018275.894717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018276.896719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018277.898740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018278.900741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018279.902745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018280.144582:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018280.144591:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018280.144595:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018280.144605:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2546 00000100:00080000:3.0:1734018280.144614:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018280.144621:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018280.144657:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334430848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018280.144690:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018280.144695:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018280.144696:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018280.144701:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2546 00000100:00080000:3.0:1734018280.144704:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018280.144708:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018280.144718:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018280.144729:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334430976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018280.144739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018280.144742:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018280.144744:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018280.144748:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2546 00000100:00080000:3.0:1734018280.144751:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018280.144779:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018280.144784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018280.144793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334431104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018280.144803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018280.144806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018280.144808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018280.144811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2546 00000100:00080000:3.0:1734018280.144814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018280.144817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734018280.144819:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334430848:192.168.201.116@tcp:250:kworker.0 00000008:00100000:3.0:1734018280.144821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018280.144831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334431232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018280.144840:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018280.144843:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018280.144845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018280.144848:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2546 00000100:00080000:3.0:1734018280.144851:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018280.144854:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018280.144858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018280.144865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334431360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018280.144874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018280.144877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018280.144879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018280.144883:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2546 00000100:00080000:3.0:1734018280.144885:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734018280.144886:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018280.144889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018280.144893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018280.144900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334431488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018280.144908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018280.144911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018280.144912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018280.144913:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018280/real 1734018280] req@ffff8800a7982680 x1818247334430848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018280.144916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2546 00000100:00080000:3.0:1734018280.144919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018280.144922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018280.144925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018280.144933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334431616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018280.144935:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334430848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018280.144945:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018280.144949:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018280.144954:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334430848:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018280.144961:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334430976:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018280.144978:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018280.144985:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018280/real 1734018280] req@ffff8800a7981880 x1818247334430976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018280.144996:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334430976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018280.145004:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018280.145007:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018280.145010:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334430976:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018280.145016:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334431104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018280.145029:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018280.145036:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018280/real 1734018280] req@ffff8800a7980a80 x1818247334431104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018280.145046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334431104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018280.145053:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018280.145056:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018280.145059:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334431104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018280.145084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334431232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018280.145100:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018280.145107:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018280/real 1734018280] req@ffff8800a7981f80 x1818247334431232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018280.145118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334431232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018280.145125:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018280.145127:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018280.145130:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334431232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018280.145135:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334431360:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018280.145147:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018280.145153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018280/real 1734018280] req@ffff8800a7982300 x1818247334431360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018280.145163:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334431360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018280.145170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018280.145173:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018280.145175:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334431360:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018280.145181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334431488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018280.145192:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018280.145199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018280/real 1734018280] req@ffff8800a7980000 x1818247334431488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018280.145215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334431488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018280.145223:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018280.145225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018280.145228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334431488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018280.145233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334431616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018280.145245:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018280.145251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018280/real 1734018280] req@ffff8800a7983480 x1818247334431616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018280.145261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334431616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018280.145268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018280.145270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018280.145273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334431616:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018280.904723:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018281.906777:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018282.908770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018283.910774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018284.912786:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018285.152622:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018285.152631:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018285.152636:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018285.152647:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2551 00000100:00080000:3.0:1734018285.152657:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018285.152665:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018285.152729:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334431744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018285.152763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018285.152767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018285.152769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018285.152774:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2551 00000100:00080000:3.0:1734018285.152777:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018285.152781:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018285.152792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018285.152803:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334431872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018285.152814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018285.152817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018285.152819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018285.152823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2551 00000100:00080000:3.0:1734018285.152826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018285.152829:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018285.152834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018285.152845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334432000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018285.152855:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018285.152858:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018285.152859:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018285.152863:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2551 00000100:00080000:3.0:1734018285.152865:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018285.152868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018285.152872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018285.152880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334432128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018285.152890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734018285.152892:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334431744:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018285.152893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018285.152894:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018285.152898:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2551 00000100:00080000:3.0:1734018285.152901:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018285.152904:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018285.152908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018285.152916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334432256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018285.152924:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018285.152927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018285.152928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018285.152932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2551 00000100:00080000:3.0:1734018285.152935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018285.152938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018285.152942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018285.152950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334432384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018285.152957:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018285.152960:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018285.152962:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018285.152965:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2551 00000100:00080000:3.0:1734018285.152970:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018285.152973:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018285.152977:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734018285.152982:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734018285.152984:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334432512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018285.153014:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018285/real 1734018285] req@ffff8800a7981c00 x1818247334431744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018285.153037:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334431744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018285.153050:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018285.153055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018285.153060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334431744:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018285.153069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334431872:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018285.153087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018285.153095:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018285/real 1734018285] req@ffff8800a7980380 x1818247334431872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018285.153106:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334431872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018285.153114:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018285.153118:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018285.153121:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334431872:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018285.153128:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334432000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018285.153143:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018285.153149:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018285/real 1734018285] req@ffff8800a7983480 x1818247334432000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018285.153159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334432000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018285.153167:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018285.153170:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018285.153173:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334432000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018285.153204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334432128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018285.153222:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018285.153228:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018285/real 1734018285] req@ffff8800a7980000 x1818247334432128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018285.153240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334432128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018285.153247:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018285.153249:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018285.153252:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334432128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018285.153258:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334432256:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018285.153271:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018285.153278:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018285/real 1734018285] req@ffff8800a7982300 x1818247334432256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018285.153288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334432256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018285.153295:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018285.153297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018285.153300:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334432256:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018285.153306:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334432384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018285.153320:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018285.153326:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018285/real 1734018285] req@ffff8800a7981f80 x1818247334432384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018285.153343:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334432384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018285.153351:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018285.153354:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018285.153357:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334432384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018285.153362:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334432512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018285.153376:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018285.153382:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018285/real 1734018285] req@ffff8800a7980a80 x1818247334432512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018285.153392:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334432512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018285.153399:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018285.153401:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018285.153404:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334432512:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018285.914754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018286.916811:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018287.918806:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018288.920722:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018289.922832:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018290.160612:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018290.160621:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018290.160625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018290.160636:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2556 00000100:00080000:3.0:1734018290.160645:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018290.160653:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018290.160690:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334432640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018290.160722:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018290.160726:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018290.160728:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018290.160733:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2556 00000100:00080000:3.0:1734018290.160736:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018290.160740:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018290.160750:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018290.160761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334432768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018290.160771:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018290.160774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018290.160776:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018290.160780:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2556 00000100:00080000:3.0:1734018290.160783:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018290.160787:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018290.160792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018290.160800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334432896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018290.160808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018290.160811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018290.160813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018290.160817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2556 00000100:00080000:3.0:1734018290.160819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018290.160822:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018290.160826:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018290.160836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334433024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018290.160837:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334432640:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018290.160844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018290.160847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018290.160849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018290.160852:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2556 00000100:00080000:3.0:1734018290.160854:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018290.160858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018290.160862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018290.160870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334433152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018290.160878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018290.160881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018290.160882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018290.160886:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2556 00000100:00080000:3.0:1734018290.160898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018290.160902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:0.0:1734018290.160903:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:3.0:1734018290.160906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018290.160914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334433280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018290.160923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018290.160926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018290.160927:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018290.160931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018290/real 1734018290] req@ffff8800a7981880 x1818247334432640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018290.160931:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2556 00000100:00080000:3.0:1734018290.160934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018290.160937:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018290.160941:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018290.160949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334433408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018290.160951:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334432640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018290.160962:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018290.160966:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018290.160971:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334432640:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018290.160979:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334432768:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018290.160995:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018290.161002:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018290/real 1734018290] req@ffff8800a7982680 x1818247334432768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018290.161013:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334432768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018290.161021:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018290.161024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018290.161028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334432768:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018290.161033:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334432896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018290.161046:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018290.161053:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018290/real 1734018290] req@ffff8800a7981500 x1818247334432896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018290.161063:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334432896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018290.161070:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018290.161072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018290.161075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334432896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018290.161105:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334433024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018290.161134:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018290.161147:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018290/real 1734018290] req@ffff8800a7983b80 x1818247334433024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018290.161164:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334433024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018290.161175:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018290.161178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018290.161184:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334433024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018290.161192:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334433152:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018290.161218:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018290.161230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018290/real 1734018290] req@ffff8800a7980700 x1818247334433152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018290.161248:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334433152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018290.161259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018290.161261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018290.161265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334433152:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018290.161271:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334433280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018290.161289:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018290.161296:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018290/real 1734018290] req@ffff8800a7981180 x1818247334433280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018290.161313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334433280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018290.161321:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018290.161323:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018290.161326:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334433280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018290.161331:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334433408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018290.161343:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018290.161349:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018290/real 1734018290] req@ffff8800a7982d80 x1818247334433408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018290.161359:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334433408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018290.161366:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018290.161368:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018290.161371:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334433408:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018290.924792:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018291.926745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018292.928776:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018293.930790:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018294.932777:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018295.168614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018295.168623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018295.168628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018295.168639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2561 00000100:00080000:3.0:1734018295.168648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018295.168656:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018295.168694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334433536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018295.168730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018295.168734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018295.168736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018295.168741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2561 00000100:00080000:3.0:1734018295.168744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018295.168748:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018295.168758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018295.168769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334433664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018295.168779:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018295.168782:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018295.168784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018295.168788:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2561 00000100:00080000:3.0:1734018295.168791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018295.168795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018295.168800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018295.168811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334433792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018295.168820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018295.168823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018295.168824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018295.168828:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2561 00000100:00080000:3.0:1734018295.168831:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018295.168834:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018295.168838:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018295.168845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334433920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018295.168856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018295.168859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018295.168860:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018295.168861:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334433536:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018295.168864:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2561 00000100:00080000:3.0:1734018295.168866:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018295.168870:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018295.168874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018295.168882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334434048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018295.168890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018295.168893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018295.168894:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018295.168898:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2561 00000100:00080000:3.0:1734018295.168900:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018295.168903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018295.168907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018295.168914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334434176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018295.168922:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018295.168925:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018295.168926:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018295.168930:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2561 00000100:00080000:3.0:1734018295.168935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018295.168938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018295.168942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018295.168949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334434304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734018295.168954:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018295.168987:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018295/real 1734018295] req@ffff8800a7983800 x1818247334433536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018295.169011:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334433536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018295.169024:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018295.169028:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018295.169034:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334433536:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018295.169042:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334433664:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018295.169061:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018295.169068:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018295/real 1734018295] req@ffff8800a7980e00 x1818247334433664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018295.169080:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334433664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018295.169088:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018295.169091:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018295.169094:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334433664:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018295.169101:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334433792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018295.169115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018295.169122:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018295/real 1734018295] req@ffff8800a7982d80 x1818247334433792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018295.169132:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334433792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018295.169140:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018295.169142:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018295.169145:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334433792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018295.169176:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334433920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018295.169193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018295.169200:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018295/real 1734018295] req@ffff8800a7981180 x1818247334433920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018295.169211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334433920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018295.169218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018295.169221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018295.169224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334433920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018295.169230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334434048:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018295.169242:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018295.169249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018295/real 1734018295] req@ffff8800a7980700 x1818247334434048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018295.169259:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334434048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018295.169266:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018295.169268:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018295.169271:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334434048:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018295.169277:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334434176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018295.169290:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018295.169297:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018295/real 1734018295] req@ffff8800a7983b80 x1818247334434176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018295.169314:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334434176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018295.169322:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018295.169325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018295.169328:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334434176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018295.169333:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334434304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018295.169347:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018295.169353:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018295/real 1734018295] req@ffff8800a7981500 x1818247334434304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018295.169363:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334434304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018295.169370:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018295.169372:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018295.169375:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334434304:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018295.934724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018296.936710:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018297.938744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018298.940756:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018299.942816:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018300.176575:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018300.176583:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018300.176588:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018300.176598:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2566 00000100:00080000:3.0:1734018300.176608:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018300.176616:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018300.176654:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334434432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018300.176690:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018300.176694:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018300.176696:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018300.176700:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2566 00000100:00080000:3.0:1734018300.176704:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018300.176708:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018300.176718:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018300.176730:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334434560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018300.176740:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018300.176743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018300.176745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018300.176749:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2566 00000100:00080000:3.0:1734018300.176752:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018300.176756:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018300.176761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018300.176769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334434688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018300.176778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018300.176781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018300.176783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018300.176786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2566 00000100:00080000:3.0:1734018300.176788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018300.176791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018300.176795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018300.176807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334434816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018300.176814:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334434432:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018300.176817:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018300.176820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018300.176822:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018300.176827:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2566 00000100:00080000:3.0:1734018300.176829:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018300.176832:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018300.176836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018300.176844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334434944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018300.176852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018300.176855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018300.176856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018300.176859:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2566 00000100:00080000:3.0:1734018300.176862:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018300.176865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018300.176869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018300.176876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334435072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018300.176884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018300.176887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018300.176888:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018300.176892:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2566 00000100:00080000:3.0:1734018300.176895:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018300.176898:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018300.176902:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734018300.176907:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:3.0:1734018300.176909:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334435200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018300.176940:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018300/real 1734018300] req@ffff8800a7982680 x1818247334434432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018300.176962:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334434432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018300.176975:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018300.176981:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018300.176986:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334434432:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018300.176995:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334434560:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018300.177013:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018300.177021:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018300/real 1734018300] req@ffff8800a7981880 x1818247334434560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018300.177032:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334434560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018300.177041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018300.177044:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018300.177047:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334434560:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018300.177053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334434688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018300.177068:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018300.177074:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018300/real 1734018300] req@ffff8800a7980a80 x1818247334434688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018300.177085:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334434688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018300.177092:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018300.177094:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018300.177097:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334434688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018300.177127:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334434816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018300.177146:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018300.177153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018300/real 1734018300] req@ffff8800a7981f80 x1818247334434816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018300.177164:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334434816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018300.177172:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018300.177174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018300.177177:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334434816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018300.177183:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334434944:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018300.177196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018300.177202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018300/real 1734018300] req@ffff8800a7982300 x1818247334434944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018300.177212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334434944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018300.177219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018300.177222:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018300.177225:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334434944:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018300.177231:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334435072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018300.177244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018300.177250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018300/real 1734018300] req@ffff8800a7980000 x1818247334435072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018300.177267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334435072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018300.177275:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018300.177277:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018300.177280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334435072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018300.177285:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334435200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018300.177299:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018300.177305:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018300/real 1734018300] req@ffff8800a7983480 x1818247334435200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018300.177315:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334435200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018300.177322:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018300.177324:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018300.177327:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334435200:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018300.944715:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018301.946842:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018302.948783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018303.950694:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018304.952771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018305.184573:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018305.184584:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018305.184589:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018305.184601:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2571 00000100:00080000:3.0:1734018305.184612:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018305.184620:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018305.184664:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334435328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018305.184710:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018305.184715:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018305.184719:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018305.184726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2571 00000100:00080000:3.0:1734018305.184731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018305.184736:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018305.184748:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018305.184766:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334435456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018305.184781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018305.184786:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018305.184790:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018305.184796:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2571 00000100:00080000:3.0:1734018305.184800:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018305.184806:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018305.184812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018305.184827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334435584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018305.184840:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334435328:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018305.184842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018305.184847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018305.184850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018305.184856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2571 00000100:00080000:3.0:1734018305.184860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018305.184865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018305.184872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018305.184887:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334435712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018305.184901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018305.184907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018305.184909:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018305.184913:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2571 00000100:00080000:3.0:1734018305.184928:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018305.184933:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:0.0:1734018305.184934:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:3.0:1734018305.184938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018305.184949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334435840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018305.184959:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018305.184962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018305.184964:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018305.184967:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018305/real 1734018305] req@ffff8800a7980380 x1818247334435328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018305.184967:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2571 00000100:00080000:3.0:1734018305.184970:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018305.184974:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018305.184978:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018305.184985:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334435968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018305.184990:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334435328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018305.184994:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018305.184997:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018305.184999:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018305.185002:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734018305.185003:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2571 00000100:00080000:3.0:1734018305.185005:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018305.185008:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:3.0:1734018305.185008:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018305.185012:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018305.185013:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334435328:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018305.185020:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334436096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018305.185022:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334435456:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018305.185041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018305.185049:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018305/real 1734018305] req@ffff8800a7981c00 x1818247334435456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018305.185061:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334435456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018305.185070:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018305.185073:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018305.185076:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334435456:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018305.185105:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334435584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018305.185119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018305.185126:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018305/real 1734018305] req@ffff8800a7983480 x1818247334435584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018305.185137:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334435584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018305.185145:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018305.185147:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018305.185150:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334435584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018305.185156:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334435712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018305.185173:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018305.185180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018305/real 1734018305] req@ffff8800a7980000 x1818247334435712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018305.185191:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334435712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018305.185198:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018305.185200:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018305.185203:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334435712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018305.185209:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334435840:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018305.185223:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018305.185230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018305/real 1734018305] req@ffff8800a7982300 x1818247334435840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018305.185240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334435840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018305.185249:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018305.185251:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018305.185254:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334435840:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018305.185260:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334435968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018305.185273:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018305.185279:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018305/real 1734018305] req@ffff8800a7981f80 x1818247334435968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018305.185297:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334435968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018305.185305:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018305.185307:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018305.185310:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334435968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018305.185316:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334436096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018305.185377:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018305.185387:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018305/real 1734018305] req@ffff8800a7980a80 x1818247334436096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018305.185397:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334436096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018305.185405:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018305.185407:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018305.185410:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334436096:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018305.954731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018306.956745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018307.958775:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018308.960753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018309.962749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018310.192597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018310.192607:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018310.192611:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018310.192623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2576 00000100:00080000:3.0:1734018310.192632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018310.192639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018310.192677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334436224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018310.192711:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018310.192716:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018310.192718:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018310.192722:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2576 00000100:00080000:3.0:1734018310.192726:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018310.192729:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018310.192739:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018310.192750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334436352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018310.192760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018310.192763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018310.192765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018310.192769:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2576 00000100:00080000:3.0:1734018310.192772:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018310.192775:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018310.192780:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018310.192788:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334436480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018310.192797:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018310.192801:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018310.192802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018310.192805:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2576 00000100:00080000:3.0:1734018310.192808:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018310.192811:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018310.192815:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018310.192822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334436224:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018310.192825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334436608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018310.192833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018310.192836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018310.192838:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018310.192841:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2576 00000100:00080000:3.0:1734018310.192843:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018310.192847:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018310.192850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018310.192858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334436736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018310.192866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018310.192869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018310.192871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018310.192875:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2576 00000100:00080000:3.0:1734018310.192887:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734018310.192890:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018310.192891:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018310.192895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018310.192903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334436864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018310.192912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018310.192915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018310.192916:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018310.192918:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018310/real 1734018310] req@ffff8800a7981880 x1818247334436224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018310.192920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2576 00000100:00080000:3.0:1734018310.192923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018310.192926:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018310.192930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018310.192937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334436992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018310.192938:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334436224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018310.192948:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018310.192953:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018310.192958:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334436224:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018310.192965:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334436352:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018310.192983:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018310.192990:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018310/real 1734018310] req@ffff8800a7982680 x1818247334436352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018310.193000:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334436352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018310.193009:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018310.193012:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018310.193016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334436352:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018310.193022:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334436480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018310.193035:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018310.193041:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018310/real 1734018310] req@ffff8800a7981500 x1818247334436480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018310.193051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334436480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018310.193058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018310.193061:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018310.193065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334436480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018310.193089:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334436608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018310.193107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018310.193114:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018310/real 1734018310] req@ffff8800a7983b80 x1818247334436608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018310.193126:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334436608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018310.193133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018310.193135:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018310.193138:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334436608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018310.193144:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334436736:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018310.193156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018310.193162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018310/real 1734018310] req@ffff8800a7980700 x1818247334436736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018310.193172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334436736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018310.193179:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018310.193181:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018310.193184:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334436736:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018310.193189:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334436864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018310.193201:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018310.193208:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018310/real 1734018310] req@ffff8800a7981180 x1818247334436864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018310.193225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334436864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018310.193233:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018310.193235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018310.193238:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334436864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018310.193288:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334436992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018310.193306:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018310.193314:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018310/real 1734018310] req@ffff8800a7982d80 x1818247334436992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018310.193325:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334436992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018310.193332:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018310.193334:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018310.193337:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334436992:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018310.964761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018311.966776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018312.968729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018313.970791:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018314.972749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018315.200580:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018315.200589:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018315.200593:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018315.200603:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2581 00000100:00080000:3.0:1734018315.200611:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018315.200619:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018315.200657:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334437120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018315.200690:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018315.200694:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018315.200696:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018315.200700:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2581 00000100:00080000:3.0:1734018315.200704:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018315.200708:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018315.200718:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018315.200729:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334437248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018315.200739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018315.200742:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018315.200743:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018315.200747:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2581 00000100:00080000:3.0:1734018315.200750:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018315.200754:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018315.200758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018315.200768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334437376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018315.200777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018315.200780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018315.200781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018315.200785:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2581 00000100:00080000:3.0:1734018315.200788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018315.200791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018315.200795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018315.200798:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334437120:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018315.200803:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334437504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018315.200812:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018315.200815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018315.200816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018315.200819:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2581 00000100:00080000:3.0:1734018315.200822:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018315.200825:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018315.200828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018315.200836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334437632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018315.200844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018315.200847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018315.200848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018315.200851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2581 00000100:00080000:3.0:1734018315.200855:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018315.200859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018315.200863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018315.200870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334437760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018315.200879:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018315.200882:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018315.200883:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018315.200886:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2581 00000100:00080000:3.0:1734018315.200889:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734018315.200891:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334437248:192.168.201.116@tcp:38:kworker.0 00000100:00080000:3.0:1734018315.200892:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018315.200896:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018315.200903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334437888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018315.200912:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334437376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018315.200935:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334437504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018315.200949:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334437632:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018315.200964:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334437760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018315.200979:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334437888:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018315.974710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018316.976765:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018317.978743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734018318.206757:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:2.0:1734018318.206805:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734018318.206824:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018318.206866:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018318.206874:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018318.206880:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018318.206889:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018318.206900:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018318.206910:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734018318.206971:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018315/real 1734018318] req@ffff8800a7981180 x1818247334437504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018318.207013:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334437504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018318.207031:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018318.207037:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018318.207044:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334437504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018318.207055:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018315/real 1734018318] req@ffff8800a7980700 x1818247334437632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018318.207077:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334437632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018318.207090:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018318.207094:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018318.207100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334437632:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018318.207152:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018315/real 1734018318] req@ffff8800a7983b80 x1818247334437760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018318.207174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334437760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018318.207187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018318.207191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018318.207197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334437760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018318.207206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018315/real 1734018318] req@ffff8800a7981500 x1818247334437888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018318.207224:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334437888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018318.207236:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018318.207240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018318.207245:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334437888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018318.207255:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018315/real 1734018318] req@ffff8800a7980e00 x1818247334437120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018318.207275:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334437120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018318.207288:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018318.207292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018318.207297:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334437120:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018318.207308:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018315/real 1734018318] req@ffff8800a7983800 x1818247334437248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018318.207327:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334437248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018318.207350:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018318.207354:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018318.207360:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334437248:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018318.207370:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018315/real 1734018318] req@ffff8800a7982d80 x1818247334437376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018318.207389:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334437376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018318.207402:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018318.207406:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018318.207411:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334437376:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018318.980751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018319.982767:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018320.208605:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018320.208613:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018320.208618:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018320.208628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2586 00000100:00080000:3.0:1734018320.208638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018320.208645:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018320.208682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334438016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018320.208718:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018320.208722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018320.208725:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018320.208730:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2586 00000100:00080000:3.0:1734018320.208733:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018320.208737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018320.208748:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018320.208759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334438144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018320.208769:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018320.208772:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018320.208774:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018320.208778:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2586 00000100:00080000:3.0:1734018320.208781:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018320.208785:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018320.208789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018320.208798:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334438272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018320.208806:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018320.208809:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018320.208810:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018320.208814:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2586 00000100:00080000:3.0:1734018320.208816:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018320.208820:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018320.208823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018320.208833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334438400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018320.208835:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334438016:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018320.208842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018320.208846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018320.208847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018320.208850:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2586 00000100:00080000:3.0:1734018320.208853:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018320.208856:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018320.208860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018320.208867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334438528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018320.208875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018320.208878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018320.208879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018320.208883:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2586 00000100:00080000:3.0:1734018320.208885:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018320.208888:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018320.208892:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018320.208900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334438656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018320.208903:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018320.208908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018320.208911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018320.208912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018320.208916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2586 00000100:00080000:3.0:1734018320.208919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018320.208923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018320.208924:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018320/real 1734018320] req@ffff8800a7982680 x1818247334438016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018320.208927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018320.208934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334438784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018320.208944:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334438016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018320.208954:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018320.208957:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018320.208962:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334438016:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018320.208970:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334438144:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018320.208987:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018320.208994:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018320/real 1734018320] req@ffff8800a7981880 x1818247334438144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018320.209006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334438144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018320.209013:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018320.209016:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018320.209019:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334438144:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018320.209025:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334438272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018320.209038:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018320.209045:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018320/real 1734018320] req@ffff8800a7980a80 x1818247334438272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018320.209055:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334438272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018320.209062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018320.209064:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018320.209067:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334438272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018320.209125:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334438400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018320.209143:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018320.209150:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018320/real 1734018320] req@ffff8800a7981f80 x1818247334438400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018320.209161:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334438400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018320.209176:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018320.209178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018320.209181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334438400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018320.209187:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334438528:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018320.209199:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018320.209205:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018320/real 1734018320] req@ffff8800a7982300 x1818247334438528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018320.209215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334438528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018320.209222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018320.209225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018320.209228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334438528:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018320.209233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334438656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018320.209245:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018320.209251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018320/real 1734018320] req@ffff8800a7980000 x1818247334438656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018320.209260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334438656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018320.209268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018320.209270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018320.209273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334438656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018320.209279:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334438784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018320.209292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018320.209298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018320/real 1734018320] req@ffff8800a7983480 x1818247334438784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018320.209308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334438784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018320.209315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018320.209317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018320.209320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334438784:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018320.984741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018321.986711:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018322.988755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018323.990734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018324.992753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018325.216616:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018325.216626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018325.216631:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018325.216642:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2591 00000100:00080000:3.0:1734018325.216650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018325.216658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018325.216695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334438912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018325.216729:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018325.216733:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018325.216735:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018325.216740:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2591 00000100:00080000:3.0:1734018325.216743:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018325.216747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018325.216757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018325.216769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334439040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018325.216779:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018325.216782:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018325.216784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018325.216788:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2591 00000100:00080000:3.0:1734018325.216791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018325.216794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018325.216799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018325.216809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334439168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018325.216818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018325.216821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018325.216822:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018325.216825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2591 00000100:00080000:3.0:1734018325.216828:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018325.216831:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018325.216835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018325.216844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334439296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018325.216847:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334438912:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018325.216853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018325.216856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018325.216857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018325.216862:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2591 00000100:00080000:3.0:1734018325.216864:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018325.216868:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018325.216872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018325.216879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334439424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018325.216887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018325.216890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018325.216892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018325.216895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2591 00000100:00080000:3.0:1734018325.216898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018325.216901:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018325.216905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018325.216912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334439552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018325.216914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018325.216919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018325.216922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018325.216924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018325.216928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2591 00000100:00080000:3.0:1734018325.216930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018325.216934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018325.216935:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018325/real 1734018325] req@ffff8800a7981c00 x1818247334438912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018325.216938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018325.216945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334439680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018325.216956:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334438912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018325.216967:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018325.216971:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018325.216975:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334438912:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018325.216983:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334439040:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018325.217044:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018325.217053:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018325/real 1734018325] req@ffff8800a7980380 x1818247334439040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018325.217073:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334439040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018325.217081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018325.217084:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018325.217087:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334439040:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018325.217093:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334439168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018325.217107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018325.217113:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018325/real 1734018325] req@ffff8800a7983480 x1818247334439168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018325.217123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334439168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018325.217131:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018325.217133:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018325.217137:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334439168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018325.217161:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334439296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018325.217174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018325.217180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018325/real 1734018325] req@ffff8800a7980000 x1818247334439296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018325.217190:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334439296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018325.217198:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018325.217200:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018325.217204:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334439296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018325.217210:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334439424:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018325.217222:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018325.217228:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018325/real 1734018325] req@ffff8800a7982300 x1818247334439424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018325.217238:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334439424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018325.217245:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018325.217247:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018325.217250:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334439424:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018325.217256:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334439552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018325.217267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018325.217273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018325/real 1734018325] req@ffff8800a7981f80 x1818247334439552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018325.217283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334439552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018325.217290:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018325.217292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018325.217295:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334439552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018325.217301:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334439680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018325.217317:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018325.217323:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018325/real 1734018325] req@ffff8800a7980a80 x1818247334439680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018325.217333:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334439680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018325.217340:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018325.217343:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018325.217346:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334439680:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018325.994733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018326.996770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018327.998754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018329.000716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018330.002783:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018330.224603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018330.224612:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018330.224617:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018330.224627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2596 00000100:00080000:3.0:1734018330.224636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018330.224643:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018330.224680:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334439808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018330.224714:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018330.224718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018330.224720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018330.224724:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2596 00000100:00080000:3.0:1734018330.224728:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018330.224732:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018330.224742:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018330.224754:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334439936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018330.224763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018330.224766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018330.224769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018330.224773:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2596 00000100:00080000:3.0:1734018330.224776:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018330.224780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018330.224785:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018330.224793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334440064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018330.224801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018330.224804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018330.224806:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018330.224809:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2596 00000100:00080000:3.0:1734018330.224811:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018330.224815:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018330.224819:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018330.224828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334440192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018330.224831:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334439808:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018330.224837:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018330.224840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018330.224842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018330.224845:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2596 00000100:00080000:3.0:1734018330.224847:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018330.224851:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018330.224855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018330.224862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334440320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018330.224871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018330.224874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018330.224876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018330.224879:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2596 00000100:00080000:3.0:1734018330.224882:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018330.224885:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018330.224889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018330.224896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334440448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018330.224897:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018330.224934:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018330.224938:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018330.224939:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018330.224943:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2596 00000100:00080000:3.0:1734018330.224946:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018330.224949:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018330.224953:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018330.224958:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018330/real 1734018330] req@ffff8800a7981880 x1818247334439808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018330.224962:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334440576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018330.224980:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334439808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018330.224992:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018330.224995:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018330.225000:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334439808:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018330.225008:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334439936:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018330.225027:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018330.225034:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018330/real 1734018330] req@ffff8800a7982680 x1818247334439936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018330.225045:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334439936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018330.225054:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018330.225057:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018330.225060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334439936:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018330.225066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334440064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018330.225079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018330.225085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018330/real 1734018330] req@ffff8800a7982d80 x1818247334440064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018330.225095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334440064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018330.225103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018330.225105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018330.225108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334440064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018330.225132:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334440192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018330.225145:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018330.225152:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018330/real 1734018330] req@ffff8800a7983800 x1818247334440192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018330.225162:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334440192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018330.225170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018330.225172:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018330.225175:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334440192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018330.225181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334440320:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018330.225194:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018330.225200:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018330/real 1734018330] req@ffff8800a7980e00 x1818247334440320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018330.225210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334440320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018330.225218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018330.225220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018330.225223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334440320:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018330.225228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334440448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018330.225240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018330.225246:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018330/real 1734018330] req@ffff8800a7981500 x1818247334440448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018330.225256:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334440448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018330.225263:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018330.225266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018330.225269:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334440448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018330.225274:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334440576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018330.225286:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018330.225293:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018330/real 1734018330] req@ffff8800a7983b80 x1818247334440576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018330.225303:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334440576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018330.225311:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018330.225313:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018330.225316:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334440576:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018331.004778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018332.006762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018333.008760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018334.010758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018335.012789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018335.232615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018335.232624:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018335.232629:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018335.232640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2601 00000100:00080000:3.0:1734018335.232650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018335.232657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018335.232694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334440704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018335.232730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018335.232734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018335.232736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018335.232740:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2601 00000100:00080000:3.0:1734018335.232743:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018335.232747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018335.232757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018335.232769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334440832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018335.232778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018335.232782:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018335.232783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018335.232787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2601 00000100:00080000:3.0:1734018335.232790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018335.232794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018335.232799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018335.232809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334440960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018335.232848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018335.232851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018335.232853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018335.232856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2601 00000100:00080000:3.0:1734018335.232859:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018335.232862:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018335.232867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018335.232876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334441088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018335.232883:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334440704:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018335.232885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018335.232888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018335.232890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018335.232894:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2601 00000100:00080000:3.0:1734018335.232897:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018335.232900:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018335.232904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018335.232912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334441216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018335.232920:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018335.232922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018335.232924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018335.232927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2601 00000100:00080000:3.0:1734018335.232931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018335.232934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018335.232938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018335.232946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334441344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018335.232952:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018335.232953:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018335.232956:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018335.232958:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018335.232961:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2601 00000100:00080000:3.0:1734018335.232964:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018335.232967:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018335.232971:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018335.232973:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018335/real 1734018335] req@ffff8800a7980700 x1818247334440704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018335.232979:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334441472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018335.232993:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334440704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018335.233003:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018335.233007:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018335.233011:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334440704:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018335.233019:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334440832:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018335.233036:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018335.233042:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018335/real 1734018335] req@ffff8800a7981180 x1818247334440832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018335.233054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334440832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018335.233061:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018335.233065:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018335.233068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334440832:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018335.233075:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334440960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018335.233088:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018335.233094:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018335/real 1734018335] req@ffff8800a7983b80 x1818247334440960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018335.233105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334440960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018335.233112:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018335.233115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018335.233119:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334440960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018335.233143:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334441088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018335.233156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018335.233162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018335/real 1734018335] req@ffff8800a7981500 x1818247334441088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018335.233172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334441088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018335.233180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018335.233182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018335.233185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334441088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018335.233190:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334441216:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018335.233206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018335.233212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018335/real 1734018335] req@ffff8800a7980e00 x1818247334441216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018335.233222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334441216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018335.233229:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018335.233231:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018335.233234:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334441216:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018335.233239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334441344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018335.233251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018335.233257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018335/real 1734018335] req@ffff8800a7983800 x1818247334441344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018335.233266:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334441344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018335.233273:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018335.233275:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018335.233279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334441344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018335.233284:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334441472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018335.233296:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018335.233302:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018335/real 1734018335] req@ffff8800a7982d80 x1818247334441472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018335.233318:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334441472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018335.233325:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018335.233327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018335.233330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334441472:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018336.014773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018337.016832:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018338.018749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018339.020799:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018340.022826:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018340.240618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018340.240627:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018340.240631:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018340.240642:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2606 00000100:00080000:3.0:1734018340.240652:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018340.240660:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018340.240700:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334441600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018340.240765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018340.240770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018340.240772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018340.240777:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2606 00000100:00080000:3.0:1734018340.240780:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018340.240784:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018340.240794:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018340.240807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334441728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018340.240818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018340.240821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018340.240822:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018340.240826:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2606 00000100:00080000:3.0:1734018340.240829:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018340.240833:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018340.240838:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018340.240846:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334441856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018340.240856:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334441600:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018340.240856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018340.240859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018340.240860:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018340.240864:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2606 00000100:00080000:3.0:1734018340.240867:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018340.240870:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018340.240874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018340.240884:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334441984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018340.240892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018340.240895:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018340.240896:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018340.240899:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2606 00000100:00080000:3.0:1734018340.240902:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018340.240905:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018340.240909:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018340.240917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334442112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018340.240924:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018340.240924:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018340.240927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018340.240929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018340.240932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2606 00000100:00080000:3.0:1734018340.240936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018340.240939:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018340.240943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018340.240945:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018340/real 1734018340] req@ffff8800a7982680 x1818247334441600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018340.240950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334442240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018340.240959:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018340.240962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018340.240964:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018340.240966:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334441600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018340.240967:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2606 00000100:00080000:3.0:1734018340.240970:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018340.240973:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734018340.240977:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734018340.240977:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018340.240980:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:3.0:1734018340.240984:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334442368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018340.240985:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334441600:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018340.240993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334441728:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018340.241010:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018340.241017:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018340/real 1734018340] req@ffff8800a7981880 x1818247334441728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018340.241029:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334441728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018340.241037:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018340.241040:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018340.241043:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334441728:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018340.241067:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334441856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018340.241081:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018340.241088:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018340/real 1734018340] req@ffff8800a7980a80 x1818247334441856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018340.241099:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334441856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018340.241106:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018340.241108:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018340.241111:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334441856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018340.241117:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334441984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018340.241129:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018340.241135:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018340/real 1734018340] req@ffff8800a7981f80 x1818247334441984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018340.241145:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334441984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018340.241152:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018340.241154:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018340.241157:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334441984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018340.241163:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334442112:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018340.241178:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018340.241184:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018340/real 1734018340] req@ffff8800a7982300 x1818247334442112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018340.241194:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334442112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018340.241202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018340.241204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018340.241207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334442112:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018340.241212:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334442240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018340.241223:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018340.241229:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018340/real 1734018340] req@ffff8800a7980000 x1818247334442240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018340.241239:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334442240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018340.241246:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018340.241248:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018340.241251:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334442240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018340.241256:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334442368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018340.241268:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018340.241275:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018340/real 1734018340] req@ffff8800a7983480 x1818247334442368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018340.241291:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334442368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018340.241299:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018340.241301:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018340.241304:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334442368:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018341.024764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018342.026796:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018343.028781:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018344.030744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018345.032753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018345.248622:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018345.248631:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018345.248636:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018345.248686:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2611 00000100:00080000:3.0:1734018345.248697:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018345.248705:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018345.248742:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334442496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018345.248778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018345.248782:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018345.248784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018345.248789:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2611 00000100:00080000:3.0:1734018345.248792:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018345.248796:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018345.248806:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018345.248817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334442624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018345.248827:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018345.248831:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018345.248833:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018345.248837:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2611 00000100:00080000:3.0:1734018345.248840:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018345.248843:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018345.248848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018345.248858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334442752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018345.248867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018345.248870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018345.248871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018345.248875:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2611 00000100:00080000:3.0:1734018345.248877:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018345.248880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018345.248884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018345.248893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334442880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018345.248898:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334442496:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018345.248903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018345.248906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018345.248908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018345.248911:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2611 00000100:00080000:3.0:1734018345.248914:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018345.248917:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018345.248920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018345.248928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334443008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018345.248936:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018345.248939:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018345.248940:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018345.248943:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2611 00000100:00080000:3.0:1734018345.248946:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018345.248949:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018345.248953:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018345.248960:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334443136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018345.248965:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018345.248968:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018345.248971:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018345.248972:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018345.248975:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2611 00000100:00100000:1.0:1734018345.248986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018345/real 1734018345] req@ffff8800a7980380 x1818247334442496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018345.248989:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018345.248992:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018345.248997:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018345.249005:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334443264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018345.249007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334442496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018345.249017:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018345.249021:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018345.249026:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334442496:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018345.249033:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334442624:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018345.249050:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018345.249057:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018345/real 1734018345] req@ffff8800a7981c00 x1818247334442624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018345.249068:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334442624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018345.249076:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018345.249079:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018345.249083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334442624:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018345.249089:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334442752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018345.249104:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018345.249110:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018345/real 1734018345] req@ffff8800a7983480 x1818247334442752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018345.249120:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334442752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018345.249128:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018345.249130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018345.249134:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334442752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018345.249159:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334442880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018345.249172:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018345.249178:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018345/real 1734018345] req@ffff8800a7980000 x1818247334442880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018345.249189:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334442880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018345.249197:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018345.249199:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018345.249202:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334442880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018345.249208:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334443008:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018345.249224:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018345.249230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018345/real 1734018345] req@ffff8800a7982300 x1818247334443008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018345.249240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334443008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018345.249247:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018345.249249:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018345.249252:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334443008:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018345.249258:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334443136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018345.249269:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018345.249275:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018345/real 1734018345] req@ffff8800a7981f80 x1818247334443136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018345.249285:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334443136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018345.249292:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018345.249294:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018345.249297:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334443136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018345.249302:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334443264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018345.249315:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018345.249321:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018345/real 1734018345] req@ffff8800a7980a80 x1818247334443264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018345.249337:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334443264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018345.249345:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018345.249347:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018345.249350:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334443264:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018346.034740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018347.036822:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018348.038756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018349.040714:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018350.042785:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018350.256638:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018350.256648:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018350.256652:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018350.256662:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2616 00000100:00080000:3.0:1734018350.256671:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018350.256678:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018350.256715:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334443392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018350.256748:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018350.256752:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018350.256754:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018350.256758:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2616 00000100:00080000:3.0:1734018350.256761:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018350.256765:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018350.256775:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018350.256787:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334443520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018350.256798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018350.256801:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018350.256803:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018350.256807:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2616 00000100:00080000:3.0:1734018350.256809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018350.256813:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018350.256818:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018350.256826:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334443648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018350.256835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018350.256839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018350.256840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018350.256843:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2616 00000100:00080000:3.0:1734018350.256846:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018350.256849:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018350.256853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018350.256856:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334443392:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018350.256862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334443776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018350.256871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018350.256874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018350.256875:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018350.256880:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2616 00000100:00080000:3.0:1734018350.256883:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018350.256886:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018350.256890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018350.256898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334443904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018350.256905:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018350.256908:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018350.256910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018350.256913:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2616 00000100:00080000:3.0:1734018350.256916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018350.256919:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018350.256923:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:3.0:1734018350.256923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018350.256930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334444032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018350.256938:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018350.256941:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018350.256943:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018350.256945:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018350/real 1734018350] req@ffff8800a7981880 x1818247334443392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018350.256946:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2616 00000100:00080000:3.0:1734018350.256949:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018350.256952:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018350.256956:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018350.256963:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334444160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018350.256966:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334443392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018350.256977:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018350.256980:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018350.256984:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334443392:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018350.256992:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334443520:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018350.257010:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018350.257016:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018350/real 1734018350] req@ffff8800a7982680 x1818247334443520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018350.257027:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334443520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018350.257035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018350.257039:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018350.257043:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334443520:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018350.257049:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334443648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018350.257063:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018350.257069:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018350/real 1734018350] req@ffff8800a7982d80 x1818247334443648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018350.257079:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334443648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018350.257087:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018350.257090:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018350.257093:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334443648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018350.257120:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334443776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018350.257133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018350.257138:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018350/real 1734018350] req@ffff8800a7983800 x1818247334443776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018350.257149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334443776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018350.257157:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018350.257159:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018350.257162:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334443776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018350.257167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334443904:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018350.257187:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018350.257193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018350/real 1734018350] req@ffff8800a7980e00 x1818247334443904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018350.257204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334443904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018350.257211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018350.257213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018350.257216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334443904:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018350.257221:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334444032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018350.257233:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018350.257239:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018350/real 1734018350] req@ffff8800a7981500 x1818247334444032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018350.257248:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334444032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018350.257255:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018350.257257:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018350.257260:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334444032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018350.257265:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334444160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018350.257278:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018350.257283:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018350/real 1734018350] req@ffff8800a7983b80 x1818247334444160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018350.257300:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334444160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018350.257307:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018350.257309:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018350.257312:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334444160:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018351.044753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018352.046704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018353.048714:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018354.050804:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018355.052727:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018355.264580:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018355.264589:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018355.264593:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018355.264603:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2621 00000100:00080000:3.0:1734018355.264612:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018355.264619:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018355.264656:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334444288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018355.264690:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018355.264694:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018355.264696:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018355.264700:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2621 00000100:00080000:3.0:1734018355.264704:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018355.264708:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018355.264717:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018355.264729:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334444416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018355.264739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018355.264742:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018355.264744:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018355.264748:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2621 00000100:00080000:3.0:1734018355.264751:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018355.264754:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018355.264759:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018355.264769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334444544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018355.264778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018355.264781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018355.264783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018355.264786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2621 00000100:00080000:3.0:1734018355.264788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018355.264792:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018355.264796:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334444288:192.168.201.116@tcp:250:kworker.0 00000008:00100000:3.0:1734018355.264796:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018355.264804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334444672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018355.264812:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018355.264815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018355.264817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018355.264820:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2621 00000100:00080000:3.0:1734018355.264822:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018355.264825:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018355.264829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018355.264837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334444800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018355.264846:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018355.264849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018355.264850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018355.264854:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2621 00000100:00080000:3.0:1734018355.264856:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018355.264860:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018355.264862:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:3.0:1734018355.264864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018355.264871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334444928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018355.264879:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018355.264882:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018355.264883:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018355/real 1734018355] req@ffff8800a7981180 x1818247334444288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018355.264883:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018355.264886:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2621 00000100:00080000:3.0:1734018355.264889:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018355.264892:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018355.264896:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018355.264903:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334444288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018355.264903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334445056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018355.264914:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018355.264917:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018355.264922:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334444288:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018355.264929:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334444416:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018355.264946:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018355.264952:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018355/real 1734018355] req@ffff8800a7980700 x1818247334444416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018355.264964:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334444416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018355.264971:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018355.264975:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018355.264978:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334444416:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018355.264984:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334444544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018355.264998:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018355.265005:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018355/real 1734018355] req@ffff8800a7983b80 x1818247334444544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018355.265015:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334444544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018355.265022:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018355.265025:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018355.265029:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334444544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018355.265053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334444672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018355.265065:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018355.265072:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018355/real 1734018355] req@ffff8800a7981500 x1818247334444672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018355.265082:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334444672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018355.265090:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018355.265092:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018355.265095:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334444672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018355.265101:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334444800:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018355.265116:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018355.265122:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018355/real 1734018355] req@ffff8800a7980e00 x1818247334444800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018355.265133:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334444800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018355.265140:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018355.265142:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018355.265145:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334444800:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018355.265151:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334444928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018355.265162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018355.265168:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018355/real 1734018355] req@ffff8800a7983800 x1818247334444928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018355.265178:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334444928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018355.265186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018355.265188:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018355.265191:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334444928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018355.265196:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334445056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018355.265208:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018355.265214:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018355/real 1734018355] req@ffff8800a7982d80 x1818247334445056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018355.265230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334445056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018355.265237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018355.265239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018355.265242:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334445056:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018356.054766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018357.056747:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018358.058743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018359.060772:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018360.062776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018360.272606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018360.272615:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018360.272620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018360.272630:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2626 00000100:00080000:3.0:1734018360.272638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018360.272646:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018360.272682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334445184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018360.272718:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018360.272722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018360.272724:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018360.272729:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2626 00000100:00080000:3.0:1734018360.272732:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018360.272736:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018360.272747:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018360.272758:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334445312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018360.272768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018360.272771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018360.272773:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018360.272777:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2626 00000100:00080000:3.0:1734018360.272779:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018360.272783:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018360.272788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018360.272796:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334445440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018360.272805:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018360.272809:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018360.272811:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018360.272814:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2626 00000100:00080000:3.0:1734018360.272817:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018360.272820:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018360.272824:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018360.272833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334445568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018360.272841:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018360.272844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018360.272845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018360.272849:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2626 00000100:00080000:3.0:1734018360.272851:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018360.272854:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018360.272858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018360.272865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334445696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018360.272875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018360.272877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018360.272879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018360.272881:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334445184:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018360.272882:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2626 00000100:00080000:3.0:1734018360.272885:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018360.272889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018360.272893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018360.272901:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334445824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018360.272909:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018360.272912:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018360.272913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018360.272917:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2626 00000100:00080000:3.0:1734018360.272919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018360.272922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018360.272926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018360.272933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334445952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018360.272981:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018360.273011:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018360/real 1734018360] req@ffff8800a7982680 x1818247334445184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018360.273034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334445184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018360.273047:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018360.273051:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018360.273056:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334445184:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018360.273065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334445312:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018360.273084:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018360.273091:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018360/real 1734018360] req@ffff8800a7981880 x1818247334445312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018360.273103:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334445312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018360.273111:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018360.273115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018360.273119:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334445312:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018360.273126:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334445440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018360.273141:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018360.273147:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018360/real 1734018360] req@ffff8800a7980a80 x1818247334445440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018360.273157:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334445440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018360.273166:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018360.273169:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018360.273172:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334445440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018360.273178:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334445568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018360.273192:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018360.273198:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018360/real 1734018360] req@ffff8800a7981f80 x1818247334445568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018360.273207:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334445568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018360.273215:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018360.273217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018360.273220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334445568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018360.273252:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334445696:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018360.273270:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018360.273277:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018360/real 1734018360] req@ffff8800a7982300 x1818247334445696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018360.273288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334445696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018360.273295:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018360.273298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018360.273301:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334445696:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018360.273307:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334445824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018360.273319:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018360.273325:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018360/real 1734018360] req@ffff8800a7980000 x1818247334445824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018360.273335:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334445824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018360.273342:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018360.273344:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018360.273347:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334445824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018360.273353:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334445952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018360.273366:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018360.273372:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018360/real 1734018360] req@ffff8800a7983480 x1818247334445952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018360.273436:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334445952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018360.273445:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018360.273448:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018360.273452:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334445952:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018361.064752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018362.066811:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018363.068706:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018364.070776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018365.072752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018365.280580:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018365.280589:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018365.280593:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018365.280604:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2631 00000100:00080000:3.0:1734018365.280614:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018365.280621:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018365.280660:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334446080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018365.280696:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018365.280701:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018365.280702:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018365.280707:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2631 00000100:00080000:3.0:1734018365.280710:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018365.280714:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018365.280724:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018365.280735:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334446208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018365.280745:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018365.280748:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018365.280750:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018365.280753:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2631 00000100:00080000:3.0:1734018365.280756:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018365.280760:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018365.280765:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018365.280775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334446336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018365.280784:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018365.280788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018365.280789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018365.280793:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2631 00000100:00080000:3.0:1734018365.280796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018365.280799:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018365.280803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018365.280811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334446464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018365.280821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018365.280823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018365.280825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018365.280828:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2631 00000100:00080000:3.0:1734018365.280831:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018365.280834:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018365.280838:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018365.280843:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334446080:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018365.280846:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334446592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018365.280854:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018365.280857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018365.280858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018365.280862:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2631 00000100:00080000:3.0:1734018365.280864:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018365.280868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018365.280872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018365.280880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334446720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018365.280888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018365.280891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018365.280893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018365.280896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2631 00000100:00080000:3.0:1734018365.280899:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018365.280902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018365.280906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018365.280913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334446848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018365.280940:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018365.280969:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018365/real 1734018365] req@ffff8800a7981c00 x1818247334446080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018365.280991:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334446080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018365.281003:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018365.281007:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018365.281012:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334446080:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018365.281020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334446208:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018365.281038:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018365.281045:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018365/real 1734018365] req@ffff8800a7980380 x1818247334446208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018365.281057:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334446208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018365.281065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018365.281069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018365.281073:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334446208:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018365.281079:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334446336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018365.281094:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018365.281101:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018365/real 1734018365] req@ffff8800a7983480 x1818247334446336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018365.281111:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334446336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018365.281119:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018365.281122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018365.281126:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334446336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018365.281156:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334446464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018365.281169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018365.281176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018365/real 1734018365] req@ffff8800a7980000 x1818247334446464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018365.281187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334446464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018365.281195:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018365.281197:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018365.281200:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334446464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018365.281206:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334446592:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018365.281223:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018365.281230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018365/real 1734018365] req@ffff8800a7982300 x1818247334446592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018365.281241:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334446592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018365.281248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018365.281250:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018365.281253:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334446592:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018365.281259:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334446720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018365.281271:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018365.281277:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018365/real 1734018365] req@ffff8800a7981f80 x1818247334446720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018365.281287:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334446720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018365.281294:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018365.281296:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018365.281299:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334446720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018365.281351:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334446848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018365.281370:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018365.281378:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018365/real 1734018365] req@ffff8800a7980a80 x1818247334446848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018365.281396:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334446848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018365.281405:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018365.281407:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018365.281410:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334446848:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018366.074788:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018367.076798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018368.078757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018369.080804:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018370.082751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018370.288599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018370.288607:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018370.288612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018370.288623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2636 00000100:00080000:3.0:1734018370.288632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018370.288639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018370.288677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334446976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018370.288712:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018370.288717:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018370.288719:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018370.288724:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2636 00000100:00080000:3.0:1734018370.288727:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018370.288731:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018370.288741:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018370.288752:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334447104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018370.288762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018370.288766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018370.288768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018370.288771:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2636 00000100:00080000:3.0:1734018370.288774:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018370.288778:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018370.288782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018370.288791:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334447232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018370.288800:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018370.288802:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018370.288804:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018370.288807:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2636 00000100:00080000:3.0:1734018370.288810:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018370.288813:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018370.288817:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018370.288825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334447360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018370.288836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018370.288839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018370.288840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018370.288844:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2636 00000100:00080000:3.0:1734018370.288847:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018370.288850:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334446976:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018370.288850:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018370.288854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018370.288862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334447488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018370.288870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018370.288873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018370.288874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018370.288878:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2636 00000100:00080000:3.0:1734018370.288880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018370.288883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018370.288887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018370.288895:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334447616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018370.288903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018370.288906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018370.288907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018370.288910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2636 00000100:00080000:3.0:1734018370.288913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018370.288916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018370.288920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018370.288927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334447744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018370.288947:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018370.288977:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018370/real 1734018370] req@ffff8800a7981880 x1818247334446976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018370.289000:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334446976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018370.289013:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018370.289017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018370.289022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334446976:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018370.289030:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334447104:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018370.289050:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018370.289057:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018370/real 1734018370] req@ffff8800a7982680 x1818247334447104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018370.289069:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334447104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018370.289077:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018370.289082:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018370.289085:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334447104:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018370.289093:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334447232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018370.289108:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018370.289114:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018370/real 1734018370] req@ffff8800a7982d80 x1818247334447232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018370.289125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334447232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018370.289133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018370.289136:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018370.289140:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334447232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018370.289170:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334447360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018370.289183:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018370.289189:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018370/real 1734018370] req@ffff8800a7983800 x1818247334447360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018370.289200:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334447360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018370.289209:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018370.289211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018370.289214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334447360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018370.289266:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334447488:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018370.289288:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018370.289296:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018370/real 1734018370] req@ffff8800a7980e00 x1818247334447488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018370.289307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334447488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018370.289315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018370.289317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018370.289320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334447488:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018370.289326:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334447616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018370.289339:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018370.289345:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018370/real 1734018370] req@ffff8800a7981500 x1818247334447616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018370.289355:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334447616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018370.289363:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018370.289365:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018370.289369:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334447616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018370.289375:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334447744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018370.289388:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018370.289394:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018370/real 1734018370] req@ffff8800a7983b80 x1818247334447744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018370.289411:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334447744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018370.289418:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018370.289421:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018370.289424:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334447744:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018371.084756:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018372.086743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018373.088787:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018374.090752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018375.092789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018375.296588:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018375.296597:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018375.296602:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018375.296612:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2641 00000100:00080000:3.0:1734018375.296620:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018375.296628:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018375.296664:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334447872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018375.296699:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018375.296703:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018375.296705:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018375.296709:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2641 00000100:00080000:3.0:1734018375.296713:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018375.296716:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018375.296727:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018375.296739:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334448000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018375.296749:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018375.296752:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018375.296754:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018375.296758:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2641 00000100:00080000:3.0:1734018375.296761:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018375.296764:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018375.296769:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018375.296779:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334448128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018375.296787:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018375.296790:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018375.296792:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018375.296795:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2641 00000100:00080000:3.0:1734018375.296797:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018375.296801:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018375.296805:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018375.296813:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334448256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018375.296814:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334447872:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018375.296823:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018375.296825:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018375.296827:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018375.296832:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2641 00000100:00080000:3.0:1734018375.296834:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018375.296837:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018375.296842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018375.296849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334448384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018375.296857:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018375.296860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018375.296861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018375.296864:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2641 00000100:00080000:3.0:1734018375.296876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018375.296880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018375.296882:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:3.0:1734018375.296884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018375.296892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334448512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018375.296900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018375.296903:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018375/real 1734018375] req@ffff8800a7980700 x1818247334447872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018375.296903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018375.296905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018375.296908:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2641 00000100:00080000:3.0:1734018375.296911:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018375.296915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018375.296919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018375.296923:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334447872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018375.296926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334448640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018375.296935:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018375.296938:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018375.296943:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334447872:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018375.296951:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334448000:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018375.296968:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018375.296975:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018375/real 1734018375] req@ffff8800a7981180 x1818247334448000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018375.296986:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334448000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018375.296994:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018375.296998:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018375.297001:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334448000:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018375.297007:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334448128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018375.297021:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018375.297027:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018375/real 1734018375] req@ffff8800a7983b80 x1818247334448128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018375.297037:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334448128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018375.297045:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018375.297047:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018375.297051:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334448128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018375.297075:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334448256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018375.297088:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018375.297094:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018375/real 1734018375] req@ffff8800a7981500 x1818247334448256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018375.297105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334448256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018375.297113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018375.297115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018375.297118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334448256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018375.297123:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334448384:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018375.297185:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018375.297193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018375/real 1734018375] req@ffff8800a7980e00 x1818247334448384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018375.297204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334448384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018375.297212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018375.297214:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018375.297218:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334448384:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018375.297223:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334448512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018375.297236:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018375.297242:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018375/real 1734018375] req@ffff8800a7983800 x1818247334448512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018375.297253:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334448512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018375.297260:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018375.297262:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018375.297265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334448512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018375.297270:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334448640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018375.297283:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018375.297289:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018375/real 1734018375] req@ffff8800a7982d80 x1818247334448640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018375.297306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334448640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018375.297314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018375.297316:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018375.297319:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334448640:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018376.094796:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018377.096657:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018378.098706:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018379.100780:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018380.102772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018380.304595:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018380.304604:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018380.304608:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018380.304618:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2646 00000100:00080000:3.0:1734018380.304627:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018380.304635:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018380.304671:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334448768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018380.304707:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018380.304712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018380.304713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018380.304718:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2646 00000100:00080000:3.0:1734018380.304721:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018380.304725:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018380.304736:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018380.304747:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334448896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018380.304757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018380.304760:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018380.304762:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018380.304766:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2646 00000100:00080000:3.0:1734018380.304769:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018380.304773:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018380.304778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018380.304786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334449024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018380.304795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018380.304798:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334448768:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018380.304798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018380.304800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018380.304803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2646 00000100:00080000:3.0:1734018380.304806:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018380.304809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018380.304813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018380.304822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334449152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018380.304831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018380.304834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018380.304835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018380.304838:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2646 00000100:00080000:3.0:1734018380.304840:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018380.304843:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018380.304847:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018380.304855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334449280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018380.304863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018380.304866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018380.304868:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018380.304871:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2646 00000100:00080000:3.0:1734018380.304874:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018380.304877:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018380.304881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018380.304888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334449408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018380.304896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018380.304899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018380.304900:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018380.304903:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2646 00000100:00080000:3.0:1734018380.304906:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018380.304909:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334448896:192.168.201.116@tcp:38:kworker.0 00000100:00080000:3.0:1734018380.304909:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018380.304913:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018380.304921:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334449536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018380.304944:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334449024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018380.304959:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334449152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018380.304975:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334449280:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018380.304990:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334449408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018380.305004:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334449536:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018381.104762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018382.106755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018383.108779:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734018383.310762:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:2.0:1734018383.310810:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734018383.310829:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018383.310865:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018383.310873:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018383.310880:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018383.310887:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018383.310893:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018383.310899:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734018383.310990:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018380/real 1734018383] req@ffff8800a7980a80 x1818247334449024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018383.311060:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334449024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018383.311077:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018383.311084:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018383.311092:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334449024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018383.311104:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018380/real 1734018383] req@ffff8800a7981f80 x1818247334449152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018383.311125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334449152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018383.311137:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018383.311141:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018383.311147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334449152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018383.311157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018380/real 1734018383] req@ffff8800a7982300 x1818247334449280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018383.311175:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334449280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018383.311188:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018383.311192:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018383.311198:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334449280:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018383.311207:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018380/real 1734018383] req@ffff8800a7980000 x1818247334449408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018383.311224:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334449408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018383.311237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018383.311241:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018383.311247:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334449408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018383.311261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018380/real 1734018383] req@ffff8800a7983480 x1818247334449536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018383.311279:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334449536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018383.311292:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018383.311296:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018383.311302:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334449536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018383.311312:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018380/real 1734018383] req@ffff8800a7982680 x1818247334448768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018383.311332:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334448768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018383.311344:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018383.311348:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018383.311353:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334448768:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018383.311364:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018380/real 1734018383] req@ffff8800a7981880 x1818247334448896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018383.311380:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334448896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018383.311392:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018383.311396:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018383.311402:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334448896:192.168.201.116@tcp:38:kworker.0 00010000:00010000:3.0:1734018384.110751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018385.112701:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018385.312579:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018385.312589:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018385.312594:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018385.312605:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2651 00000100:00080000:3.0:1734018385.312614:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018385.312621:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018385.312659:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334449664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018385.312695:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018385.312699:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018385.312701:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018385.312705:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2651 00000100:00080000:3.0:1734018385.312708:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018385.312712:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018385.312723:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018385.312735:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334449792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018385.312744:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018385.312748:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018385.312750:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018385.312754:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2651 00000100:00080000:3.0:1734018385.312756:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018385.312760:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018385.312765:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018385.312774:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334449920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018385.312783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018385.312787:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018385.312788:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018385.312791:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2651 00000100:00080000:3.0:1734018385.312794:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018385.312797:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018385.312801:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018385.312807:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334449664:192.168.201.116@tcp:250:kworker.0 00000100:00100000:3.0:1734018385.312809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334450048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018385.312818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018385.312821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018385.312823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018385.312826:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2651 00000100:00080000:3.0:1734018385.312828:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018385.312831:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018385.312835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018385.312843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334450176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018385.312851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018385.312854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018385.312855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018385.312859:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2651 00000100:00080000:3.0:1734018385.312871:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018385.312875:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018385.312876:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:3.0:1734018385.312879:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018385.312887:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334450304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018385.312895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018385.312898:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018385/real 1734018385] req@ffff8800a7980380 x1818247334449664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018385.312898:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018385.312900:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018385.312903:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2651 00000100:00080000:3.0:1734018385.312906:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018385.312909:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018385.312913:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018385.312917:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334449664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018385.312920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334450432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018385.312929:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018385.312932:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018385.312937:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334449664:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018385.312944:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334449792:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018385.312961:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018385.313001:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018385/real 1734018385] req@ffff8800a7981c00 x1818247334449792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018385.313013:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334449792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018385.313021:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018385.313024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018385.313027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334449792:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018385.313034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334449920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018385.313051:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018385.313057:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018385/real 1734018385] req@ffff8800a7981880 x1818247334449920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018385.313067:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334449920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018385.313076:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018385.313079:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018385.313082:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334449920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018385.313105:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334450048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018385.313118:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018385.313125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018385/real 1734018385] req@ffff8800a7982680 x1818247334450048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018385.313135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334450048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018385.313142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018385.313145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018385.313148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334450048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018385.313153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334450176:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018385.313165:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018385.313171:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018385/real 1734018385] req@ffff8800a7983480 x1818247334450176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018385.313180:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334450176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018385.313188:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018385.313190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018385.313193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334450176:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018385.313199:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334450304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018385.313211:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018385.313217:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018385/real 1734018385] req@ffff8800a7980000 x1818247334450304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018385.313227:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334450304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018385.313234:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018385.313236:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018385.313239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334450304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018385.313244:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334450432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018385.313256:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018385.313262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018385/real 1734018385] req@ffff8800a7982300 x1818247334450432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018385.313272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334450432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018385.313280:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018385.313282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018385.313285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334450432:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018386.114755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018387.116778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018388.118749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018389.120782:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018390.122704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018390.320604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018390.320613:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018390.320618:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018390.320629:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2656 00000100:00080000:3.0:1734018390.320638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018390.320646:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018390.320685:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334450560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018390.320722:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018390.320726:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018390.320728:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018390.320733:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2656 00000100:00080000:3.0:1734018390.320736:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018390.320740:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018390.320751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018390.320762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334450688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018390.320772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018390.320775:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018390.320777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018390.320781:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2656 00000100:00080000:3.0:1734018390.320784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018390.320788:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018390.320792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018390.320801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334450816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018390.320809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018390.320812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018390.320814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018390.320817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2656 00000100:00080000:3.0:1734018390.320820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018390.320823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018390.320827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018390.320837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334450944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018390.320839:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334450560:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018390.320846:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018390.320849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018390.320851:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018390.320854:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2656 00000100:00080000:3.0:1734018390.320856:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018390.320859:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018390.320864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018390.320871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334451072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018390.320918:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018390.320922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018390.320923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018390.320927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2656 00000100:00080000:3.0:1734018390.320930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018390.320934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018390.320939:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:3.0:1734018390.320939:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018390.320948:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334451200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018390.320957:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018390.320960:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018390.320962:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018390/real 1734018390] req@ffff8800a7981f80 x1818247334450560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018390.320962:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018390.320965:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2656 00000100:00080000:3.0:1734018390.320968:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018390.320971:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018390.320975:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018390.320982:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334450560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018390.320983:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334451328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018390.320993:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018390.320997:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018390.321002:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334450560:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018390.321010:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334450688:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018390.321028:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018390.321035:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018390/real 1734018390] req@ffff8800a7980a80 x1818247334450688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018390.321047:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334450688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018390.321055:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018390.321057:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018390.321061:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334450688:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018390.321067:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334450816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018390.321080:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018390.321086:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018390/real 1734018390] req@ffff8800a7982d80 x1818247334450816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018390.321096:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334450816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018390.321104:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018390.321106:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018390.321110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334450816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018390.321136:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334450944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018390.321150:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018390.321156:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018390/real 1734018390] req@ffff8800a7983800 x1818247334450944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018390.321167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334450944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018390.321174:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018390.321176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018390.321179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334450944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018390.321184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334451072:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018390.321196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018390.321202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018390/real 1734018390] req@ffff8800a7980e00 x1818247334451072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018390.321212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334451072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018390.321219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018390.321221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018390.321224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334451072:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018390.321230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334451200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018390.321245:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018390.321251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018390/real 1734018390] req@ffff8800a7981500 x1818247334451200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018390.321261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334451200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018390.321268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018390.321270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018390.321273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334451200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018390.321279:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334451328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018390.321290:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018390.321296:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018390/real 1734018390] req@ffff8800a7983b80 x1818247334451328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018390.321306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334451328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018390.321313:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018390.321315:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018390.321318:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334451328:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018391.124745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018392.126784:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018393.128733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018394.130755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018395.132762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018395.328610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018395.328619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018395.328624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018395.328635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2661 00000100:00080000:3.0:1734018395.328643:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018395.328651:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018395.328689:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334451456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018395.328724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018395.328728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018395.328731:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018395.328735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2661 00000100:00080000:3.0:1734018395.328739:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018395.328742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018395.328753:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018395.328765:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334451584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018395.328775:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018395.328778:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018395.328780:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018395.328784:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2661 00000100:00080000:3.0:1734018395.328787:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018395.328791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018395.328822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018395.328833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334451712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018395.328843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018395.328846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018395.328848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018395.328851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2661 00000100:00080000:3.0:1734018395.328854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018395.328857:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018395.328862:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334451456:192.168.201.116@tcp:250:kworker.0 00000008:00100000:3.0:1734018395.328862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018395.328870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334451840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018395.328880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018395.328883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018395.328884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018395.328888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2661 00000100:00080000:3.0:1734018395.328890:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018395.328893:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018395.328897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018395.328905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334451968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018395.328913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018395.328916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018395.328917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018395.328920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2661 00000100:00080000:3.0:1734018395.328923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018395.328926:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018395.328930:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:3.0:1734018395.328930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018395.328939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334452096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018395.328947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018395.328950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018395.328952:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018395/real 1734018395] req@ffff8800a7981180 x1818247334451456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018395.328952:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018395.328955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2661 00000100:00080000:3.0:1734018395.328958:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018395.328962:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018395.328966:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018395.328973:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334452224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018395.328980:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334451456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018395.328990:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018395.328993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018395.328998:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334451456:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018395.329005:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334451584:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018395.329021:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018395.329029:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018395/real 1734018395] req@ffff8800a7980700 x1818247334451584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018395.329040:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334451584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018395.329047:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018395.329050:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018395.329053:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334451584:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018395.329059:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334451712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018395.329072:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018395.329078:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018395/real 1734018395] req@ffff8800a7983b80 x1818247334451712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018395.329089:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334451712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018395.329096:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018395.329099:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018395.329102:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334451712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018395.329127:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334451840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018395.329140:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018395.329146:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018395/real 1734018395] req@ffff8800a7981500 x1818247334451840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018395.329157:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334451840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018395.329164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018395.329166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018395.329169:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334451840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018395.329174:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334451968:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018395.329186:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018395.329192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018395/real 1734018395] req@ffff8800a7980e00 x1818247334451968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018395.329202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334451968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018395.329209:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018395.329212:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018395.329215:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334451968:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018395.329220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334452096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018395.329235:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018395.329242:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018395/real 1734018395] req@ffff8800a7983800 x1818247334452096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018395.329252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334452096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018395.329259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018395.329261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018395.329264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334452096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018395.329269:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334452224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018395.329281:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018395.329287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018395/real 1734018395] req@ffff8800a7982d80 x1818247334452224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018395.329297:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334452224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018395.329304:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018395.329306:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018395.329309:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334452224:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018396.134752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018397.136757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018398.138804:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018399.140769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018400.142751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018400.336614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018400.336622:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018400.336627:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018400.336637:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2666 00000100:00080000:3.0:1734018400.336646:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018400.336653:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018400.336690:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334452352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018400.336765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018400.336770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018400.336772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018400.336777:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2666 00000100:00080000:3.0:1734018400.336781:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018400.336785:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018400.336796:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018400.336810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334452480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018400.336820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018400.336824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018400.336826:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018400.336830:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2666 00000100:00080000:3.0:1734018400.336832:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018400.336835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018400.336840:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018400.336848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334452608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018400.336857:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018400.336860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018400.336861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018400.336864:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2666 00000100:00080000:3.0:1734018400.336867:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018400.336870:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018400.336874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018400.336884:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334452736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018400.336886:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334452352:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018400.336893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018400.336896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018400.336898:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018400.336903:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2666 00000100:00080000:3.0:1734018400.336905:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018400.336908:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018400.336912:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018400.336920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334452864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018400.336928:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018400.336931:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018400.336932:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018400.336935:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2666 00000100:00080000:3.0:1734018400.336938:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018400.336941:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018400.336945:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018400.336953:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334452992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018400.336954:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018400.336960:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018400.336963:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018400.336965:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018400.336968:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2666 00000100:00080000:3.0:1734018400.336971:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018400.336975:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018400.336977:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018400/real 1734018400] req@ffff8800a7980a80 x1818247334452352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018400.336979:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018400.336987:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334453120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018400.337005:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334452352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018400.337015:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018400.337019:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018400.337023:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334452352:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018400.337031:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334452480:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018400.337047:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018400.337054:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018400/real 1734018400] req@ffff8800a7981f80 x1818247334452480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018400.337065:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334452480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018400.337073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018400.337075:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018400.337079:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334452480:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018400.337084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334452608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018400.337098:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018400.337104:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018400/real 1734018400] req@ffff8800a7982300 x1818247334452608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018400.337114:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334452608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018400.337121:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018400.337124:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018400.337127:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334452608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018400.337152:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334452736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018400.337165:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018400.337171:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018400/real 1734018400] req@ffff8800a7980000 x1818247334452736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018400.337182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334452736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018400.337189:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018400.337191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018400.337194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334452736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018400.337199:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334452864:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018400.337211:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018400.337218:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018400/real 1734018400] req@ffff8800a7983480 x1818247334452864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018400.337227:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334452864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018400.337235:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018400.337237:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018400.337240:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334452864:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018400.337245:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334452992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018400.337266:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018400.337272:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018400/real 1734018400] req@ffff8800a7982680 x1818247334452992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018400.337282:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334452992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018400.337289:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018400.337291:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018400.337294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334452992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018400.337300:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334453120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018400.337312:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018400.337318:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018400/real 1734018400] req@ffff8800a7981880 x1818247334453120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018400.337328:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334453120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018400.337335:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018400.337337:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018400.337340:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334453120:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018401.144782:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018402.146766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018403.148767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018404.150836:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018405.152778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018405.344597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018405.344606:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018405.344611:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018405.344661:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2671 00000100:00080000:3.0:1734018405.344671:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018405.344679:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018405.344718:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334453248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018405.344753:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018405.344757:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018405.344759:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018405.344763:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2671 00000100:00080000:3.0:1734018405.344767:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018405.344771:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018405.344781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018405.344793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334453376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018405.344803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018405.344806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018405.344808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018405.344812:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2671 00000100:00080000:3.0:1734018405.344815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018405.344819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018405.344824:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018405.344834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334453504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018405.344842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018405.344845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018405.344847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018405.344851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2671 00000100:00080000:3.0:1734018405.344853:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018405.344857:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018405.344860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018405.344869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334453632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018405.344873:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334453248:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018405.344878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018405.344881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018405.344882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018405.344885:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2671 00000100:00080000:3.0:1734018405.344888:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018405.344891:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018405.344895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018405.344902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334453760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018405.344911:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018405.344914:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018405.344915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018405.344919:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2671 00000100:00080000:3.0:1734018405.344921:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018405.344924:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018405.344928:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018405.344936:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334453888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018405.344939:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018405.344943:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018405.344946:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018405.344948:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018405.344951:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2671 00000100:00100000:1.0:1734018405.344962:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018405/real 1734018405] req@ffff8800a7981c00 x1818247334453248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018405.344963:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018405.344967:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018405.344971:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018405.344979:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334454016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018405.344990:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334453248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018405.345000:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018405.345004:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018405.345009:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334453248:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018405.345016:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334453376:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018405.345033:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018405.345040:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018405/real 1734018405] req@ffff8800a7980380 x1818247334453376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018405.345051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334453376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018405.345058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018405.345061:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018405.345064:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334453376:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018405.345070:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334453504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018405.345083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018405.345090:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018405/real 1734018405] req@ffff8800a7981880 x1818247334453504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018405.345100:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334453504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018405.345107:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018405.345110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018405.345113:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334453504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018405.345137:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334453632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018405.345151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018405.345157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018405/real 1734018405] req@ffff8800a7982680 x1818247334453632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018405.345168:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334453632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018405.345175:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018405.345177:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018405.345181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334453632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018405.345186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334453760:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018405.345198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018405.345204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018405/real 1734018405] req@ffff8800a7983480 x1818247334453760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018405.345214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334453760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018405.345221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018405.345224:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018405.345227:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334453760:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018405.345232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334453888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018405.345248:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018405.345254:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018405/real 1734018405] req@ffff8800a7980000 x1818247334453888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018405.345264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334453888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018405.345271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018405.345274:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018405.345276:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334453888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018405.345282:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334454016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018405.345294:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018405.345300:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018405/real 1734018405] req@ffff8800a7982300 x1818247334454016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018405.345309:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334454016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018405.345316:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018405.345319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018405.345322:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334454016:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018406.154782:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018407.156802:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018408.158795:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018409.160802:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018410.162760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018410.352660:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018410.352669:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018410.352673:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018410.352684:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2676 00000100:00080000:3.0:1734018410.352694:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018410.352701:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018410.352739:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334454144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018410.352775:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018410.352779:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018410.352781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018410.352785:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2676 00000100:00080000:3.0:1734018410.352789:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018410.352792:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018410.352802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018410.352814:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334454272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018410.352823:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018410.352827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018410.352828:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018410.352833:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2676 00000100:00080000:3.0:1734018410.352835:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018410.352839:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018410.352844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018410.352853:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334454400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018410.352862:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018410.352865:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018410.352867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018410.352870:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2676 00000100:00080000:3.0:1734018410.352873:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018410.352876:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018410.352879:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018410.352889:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334454528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018410.352890:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334454144:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018410.352897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018410.352900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018410.352902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018410.352905:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2676 00000100:00080000:3.0:1734018410.352907:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018410.352911:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018410.352915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018410.352922:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334454656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018410.352930:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018410.352933:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018410.352935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018410.352938:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2676 00000100:00080000:3.0:1734018410.352942:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018410.352945:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018410.352949:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018410.352956:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334454784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018410.352959:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018410.352964:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018410.352967:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018410.352969:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018410.352972:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2676 00000100:00080000:3.0:1734018410.352976:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018410.352979:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018410.352981:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018410/real 1734018410] req@ffff8800a7981f80 x1818247334454144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018410.352983:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018410.352991:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334454912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018410.353008:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334454144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018410.353019:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018410.353022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018410.353027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334454144:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018410.353034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334454272:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018410.353051:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018410.353058:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018410/real 1734018410] req@ffff8800a7980a80 x1818247334454272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018410.353069:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334454272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018410.353077:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018410.353079:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018410.353082:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334454272:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018410.353088:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334454400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018410.353102:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018410.353108:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018410/real 1734018410] req@ffff8800a7982d80 x1818247334454400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018410.353118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334454400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018410.353125:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018410.353129:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018410.353132:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334454400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018410.353159:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334454528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018410.353172:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018410.353178:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018410/real 1734018410] req@ffff8800a7983800 x1818247334454528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018410.353189:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334454528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018410.353196:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018410.353198:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018410.353201:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334454528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018410.353206:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334454656:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018410.353218:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018410.353224:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018410/real 1734018410] req@ffff8800a7980e00 x1818247334454656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018410.353234:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334454656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018410.353241:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018410.353243:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018410.353246:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334454656:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018410.353252:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334454784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018410.353267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018410.353273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018410/real 1734018410] req@ffff8800a7981500 x1818247334454784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018410.353284:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334454784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018410.353291:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018410.353293:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018410.353296:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334454784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018410.353301:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334454912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018410.353313:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018410.353319:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018410/real 1734018410] req@ffff8800a7983b80 x1818247334454912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018410.353329:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334454912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018410.353336:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018410.353338:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018410.353341:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334454912:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018411.164822:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018412.166808:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018413.168782:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018414.170797:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018415.172774:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018415.360608:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018415.360617:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018415.360622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018415.360632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2681 00000100:00080000:3.0:1734018415.360642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018415.360650:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018415.360688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334455040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018415.360725:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018415.360729:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018415.360731:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018415.360735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2681 00000100:00080000:3.0:1734018415.360738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018415.360742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018415.360752:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018415.360763:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334455168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018415.360773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018415.360776:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018415.360777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018415.360782:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2681 00000100:00080000:3.0:1734018415.360784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018415.360788:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018415.360793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018415.360803:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334455296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018415.360812:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018415.360815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018415.360817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018415.360821:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2681 00000100:00080000:3.0:1734018415.360823:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018415.360826:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018415.360830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018415.360839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334455424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018415.360842:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334455040:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018415.360848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018415.360850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018415.360852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018415.360855:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2681 00000100:00080000:3.0:1734018415.360858:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018415.360861:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018415.360865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018415.360873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334455552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018415.360881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018415.360884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018415.360885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018415.360888:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2681 00000100:00080000:3.0:1734018415.360891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018415.360894:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018415.360898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018415.360906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334455680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018415.360911:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018415.360914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018415.360917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018415.360919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018415.360922:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2681 00000100:00100000:1.0:1734018415.360933:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018415/real 1734018415] req@ffff8800a7980700 x1818247334455040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018415.360934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018415.360938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018415.360942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018415.360950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334455808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018415.360961:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334455040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018415.360971:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018415.360974:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018415.360979:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334455040:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018415.360986:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334455168:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018415.361003:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018415.361010:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018415/real 1734018415] req@ffff8800a7981180 x1818247334455168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018415.361021:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334455168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018415.361029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018415.361032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018415.361035:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334455168:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018415.361041:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334455296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018415.361054:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018415.361061:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018415/real 1734018415] req@ffff8800a7983b80 x1818247334455296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018415.361071:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334455296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018415.361078:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018415.361081:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018415.361084:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334455296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018415.361108:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334455424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018415.361122:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018415.361128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018415/real 1734018415] req@ffff8800a7981500 x1818247334455424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018415.361138:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334455424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018415.361146:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018415.361148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018415.361151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334455424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018415.361156:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334455552:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018415.361169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018415.361175:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018415/real 1734018415] req@ffff8800a7980e00 x1818247334455552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018415.361184:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334455552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018415.361192:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018415.361194:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018415.361197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334455552:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018415.361202:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334455680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018415.361217:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018415.361224:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018415/real 1734018415] req@ffff8800a7983800 x1818247334455680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018415.361234:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334455680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018415.361241:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018415.361243:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018415.361246:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334455680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018415.361251:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334455808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018415.361263:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018415.361269:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018415/real 1734018415] req@ffff8800a7982d80 x1818247334455808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018415.361279:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334455808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018415.361286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018415.361288:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018415.361291:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334455808:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018416.174723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018417.176807:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018418.178776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018419.180721:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018420.182774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018420.368588:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018420.368597:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018420.368601:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018420.368612:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2687 00000100:00080000:3.0:1734018420.368621:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018420.368629:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018420.368666:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334455936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018420.368702:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018420.368706:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018420.368709:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018420.368713:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2687 00000100:00080000:3.0:1734018420.368716:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018420.368720:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018420.368731:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018420.368741:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334456064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018420.368751:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018420.368755:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018420.368757:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018420.368760:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2687 00000100:00080000:3.0:1734018420.368763:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018420.368767:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018420.368771:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018420.368780:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334456192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018420.368788:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018420.368791:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018420.368793:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018420.368796:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2687 00000100:00080000:3.0:1734018420.368798:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018420.368802:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018420.368806:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018420.368815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334456320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018420.368823:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334455936:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018420.368825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018420.368828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018420.368829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018420.368833:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2687 00000100:00080000:3.0:1734018420.368835:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018420.368838:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018420.368842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018420.368850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334456448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018420.368858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018420.368860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018420.368862:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018420.368865:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2687 00000100:00080000:3.0:1734018420.368868:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018420.368871:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018420.368875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018420.368882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334456576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018420.368890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734018420.368892:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018420.368893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018420.368894:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018420.368897:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2687 00000100:00080000:3.0:1734018420.368901:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018420.368904:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018420.368908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018420.368914:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018420/real 1734018420] req@ffff8800a7980a80 x1818247334455936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018420.368916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334456704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018420.368942:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334455936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018420.368953:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018420.368957:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018420.368961:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334455936:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018420.368968:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334456064:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018420.368984:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018420.368992:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018420/real 1734018420] req@ffff8800a7981f80 x1818247334456064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018420.369003:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334456064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018420.369010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018420.369013:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018420.369016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334456064:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018420.369022:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334456192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018420.369035:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018420.369041:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018420/real 1734018420] req@ffff8800a7982300 x1818247334456192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018420.369051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334456192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018420.369058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018420.369062:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018420.369065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334456192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018420.369090:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334456320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018420.369103:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018420.369110:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018420/real 1734018420] req@ffff8800a7980000 x1818247334456320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018420.369120:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334456320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018420.369128:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018420.369130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018420.369133:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334456320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018420.369138:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334456448:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018420.369150:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018420.369156:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018420/real 1734018420] req@ffff8800a7983480 x1818247334456448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018420.369166:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334456448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018420.369173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018420.369175:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018420.369178:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334456448:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018420.369184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334456576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018420.369200:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018420.369206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018420/real 1734018420] req@ffff8800a7982680 x1818247334456576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018420.369217:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334456576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018420.369224:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018420.369226:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018420.369229:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334456576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018420.369234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334456704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018420.369246:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018420.369252:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018420/real 1734018420] req@ffff8800a7981880 x1818247334456704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018420.369262:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334456704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018420.369269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018420.369271:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018420.369274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334456704:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018421.184733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018422.186800:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018423.188768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018424.190736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018425.192729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018425.376564:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018425.376573:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018425.376578:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018425.376589:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2692 00000100:00080000:3.0:1734018425.376597:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018425.376605:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018425.376643:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334456832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018425.376679:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018425.376683:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018425.376685:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018425.376689:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2692 00000100:00080000:3.0:1734018425.376692:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018425.376696:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018425.376707:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018425.376718:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334456960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018425.376728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018425.376732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018425.376733:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018425.376737:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2692 00000100:00080000:3.0:1734018425.376740:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018425.376743:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018425.376748:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018425.376759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334457088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018425.376767:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018425.376770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018425.376772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018425.376775:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2692 00000100:00080000:3.0:1734018425.376778:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018425.376781:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018425.376785:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018425.376794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334457216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018425.376796:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334456832:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018425.376803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018425.376806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018425.376807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018425.376812:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2692 00000100:00080000:3.0:1734018425.376815:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018425.376818:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018425.376822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018425.376830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334457344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018425.376838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018425.376841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018425.376842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018425.376845:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2692 00000100:00080000:3.0:1734018425.376848:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018425.376851:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018425.376855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018425.376862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334457472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018425.376865:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018425.376870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018425.376873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018425.376875:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018425.376878:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2692 00000100:00080000:3.0:1734018425.376882:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018425.376885:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018425.376887:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018425/real 1734018425] req@ffff8800a7980380 x1818247334456832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018425.376889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018425.376897:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334457600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018425.376914:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334456832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018425.376925:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018425.376928:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018425.376932:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334456832:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018425.376940:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334456960:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018425.376956:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018425.376963:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018425/real 1734018425] req@ffff8800a7981c00 x1818247334456960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018425.376974:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334456960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018425.376982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018425.376984:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018425.376988:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334456960:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018425.376993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334457088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018425.377007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018425.377013:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018425/real 1734018425] req@ffff8800a7981880 x1818247334457088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018425.377023:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334457088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018425.377030:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018425.377034:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018425.377037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334457088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018425.377061:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334457216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018425.377074:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018425.377080:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018425/real 1734018425] req@ffff8800a7982680 x1818247334457216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018425.377090:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334457216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018425.377098:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018425.377100:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018425.377103:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334457216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018425.377108:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334457344:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018425.377120:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018425.377126:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018425/real 1734018425] req@ffff8800a7983480 x1818247334457344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018425.377136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334457344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018425.377143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018425.377145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018425.377148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334457344:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018425.377153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334457472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018425.377174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018425.377180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018425/real 1734018425] req@ffff8800a7980000 x1818247334457472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018425.377191:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334457472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018425.377198:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018425.377200:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018425.377203:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334457472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018425.377208:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334457600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018425.377220:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018425.377226:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018425/real 1734018425] req@ffff8800a7982300 x1818247334457600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018425.377236:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334457600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018425.377243:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018425.377245:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018425.377248:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334457600:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018426.194763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018427.196773:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018428.198729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018429.200738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018430.202748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018430.384602:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018430.384610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018430.384615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018430.384625:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2697 00000100:00080000:3.0:1734018430.384634:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018430.384641:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018430.384677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334457728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018430.384714:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018430.384718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018430.384720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018430.384724:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2697 00000100:00080000:3.0:1734018430.384727:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018430.384731:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018430.384741:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018430.384752:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334457856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018430.384762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018430.384765:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018430.384767:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018430.384771:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2697 00000100:00080000:3.0:1734018430.384774:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018430.384777:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018430.384782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018430.384791:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334457984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018430.384799:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018430.384802:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018430.384804:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018430.384807:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2697 00000100:00080000:3.0:1734018430.384809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018430.384813:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018430.384817:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018430.384826:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334458112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018430.384831:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334457728:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018430.384835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018430.384838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018430.384840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018430.384843:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2697 00000100:00080000:3.0:1734018430.384846:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018430.384849:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018430.384853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018430.384860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334458240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018430.384869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018430.384872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018430.384873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018430.384877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2697 00000100:00080000:3.0:1734018430.384879:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018430.384882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018430.384886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018430.384894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334458368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018430.384899:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018430.384901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018430.384904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018430.384906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018430.384909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2697 00000100:00080000:3.0:1734018430.384912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018430.384915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018430.384919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018430.384922:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018430/real 1734018430] req@ffff8800a7981f80 x1818247334457728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018430.384926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334458496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018430.384950:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334457728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018430.384960:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018430.384963:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018430.384968:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334457728:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018430.384975:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334457856:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018430.384991:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018430.384998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018430/real 1734018430] req@ffff8800a7980a80 x1818247334457856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018430.385009:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334457856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018430.385016:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018430.385019:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018430.385022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334457856:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018430.385028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334457984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018430.385041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018430.385047:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018430/real 1734018430] req@ffff8800a7982d80 x1818247334457984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018430.385057:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334457984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018430.385065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018430.385068:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018430.385071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334457984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018430.385099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334458112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018430.385112:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018430.385118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018430/real 1734018430] req@ffff8800a7983800 x1818247334458112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018430.385129:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334458112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018430.385136:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018430.385138:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018430.385141:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334458112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018430.385146:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334458240:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018430.385158:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018430.385164:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018430/real 1734018430] req@ffff8800a7980e00 x1818247334458240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018430.385174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334458240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018430.385181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018430.385183:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018430.385186:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334458240:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018430.385191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334458368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018430.385259:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018430.385267:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018430/real 1734018430] req@ffff8800a7981500 x1818247334458368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018430.385278:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334458368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018430.385286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018430.385289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018430.385292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334458368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018430.385297:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334458496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018430.385310:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018430.385316:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018430/real 1734018430] req@ffff8800a7983b80 x1818247334458496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018430.385326:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334458496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018430.385333:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018430.385335:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018430.385338:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334458496:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018431.204793:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018432.206742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018433.208752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018434.210757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018435.212778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018435.392606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018435.392615:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018435.392620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018435.392631:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2702 00000100:00080000:3.0:1734018435.392640:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018435.392647:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018435.392685:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334458624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018435.392722:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018435.392726:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018435.392727:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018435.392732:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2702 00000100:00080000:3.0:1734018435.392735:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018435.392739:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018435.392750:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018435.392761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334458752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018435.392770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018435.392774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018435.392775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018435.392779:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2702 00000100:00080000:3.0:1734018435.392782:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018435.392786:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018435.392790:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018435.392800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334458880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018435.392810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018435.392813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018435.392814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018435.392818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2702 00000100:00080000:3.0:1734018435.392820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018435.392823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018435.392827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018435.392835:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334459008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018435.392838:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334458624:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018435.392844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018435.392847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018435.392848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018435.392851:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2702 00000100:00080000:3.0:1734018435.392854:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018435.392857:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018435.392861:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018435.392868:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334459136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018435.392876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018435.392879:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018435.392880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018435.392884:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2702 00000100:00080000:3.0:1734018435.392887:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018435.392891:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018435.392895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018435.392903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334459264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018435.392906:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018435.392910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018435.392913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018435.392915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018435.392918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2702 00000100:00080000:3.0:1734018435.392923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018435.392926:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018435.392927:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018435/real 1734018435] req@ffff8800a7981180 x1818247334458624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018435.392930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018435.392938:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334459392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018435.392955:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334458624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018435.392966:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018435.392969:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018435.392974:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334458624:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018435.392981:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334458752:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018435.392998:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018435.393005:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018435/real 1734018435] req@ffff8800a7980700 x1818247334458752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018435.393016:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334458752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018435.393024:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018435.393027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018435.393030:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334458752:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018435.393036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334458880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018435.393049:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018435.393055:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018435/real 1734018435] req@ffff8800a7983b80 x1818247334458880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018435.393065:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334458880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018435.393073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018435.393076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018435.393079:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334458880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018435.393104:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334459008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018435.393166:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018435.393175:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018435/real 1734018435] req@ffff8800a7981500 x1818247334459008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018435.393186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334459008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018435.393194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018435.393196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018435.393199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334459008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018435.393205:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334459136:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018435.393219:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018435.393226:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018435/real 1734018435] req@ffff8800a7980e00 x1818247334459136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018435.393236:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334459136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018435.393244:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018435.393246:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018435.393249:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334459136:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018435.393255:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334459264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018435.393270:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018435.393276:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018435/real 1734018435] req@ffff8800a7983800 x1818247334459264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018435.393286:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334459264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018435.393294:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018435.393296:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018435.393299:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334459264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018435.393304:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334459392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018435.393316:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018435.393322:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018435/real 1734018435] req@ffff8800a7982d80 x1818247334459392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018435.393332:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334459392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018435.393339:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018435.393341:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018435.393344:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334459392:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018436.214758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018437.216778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018438.218780:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018439.220743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018440.222770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018440.400607:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018440.400617:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018440.400622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018440.400633:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2707 00000100:00080000:3.0:1734018440.400642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018440.400650:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018440.400688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334459520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018440.400724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018440.400728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018440.400730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018440.400734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2707 00000100:00080000:3.0:1734018440.400738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018440.400741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018440.400751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018440.400763:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334459648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018440.400772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018440.400776:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018440.400777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018440.400781:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2707 00000100:00080000:3.0:1734018440.400784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018440.400788:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018440.400792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018440.400800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334459776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018440.400809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018440.400812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018440.400813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018440.400817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2707 00000100:00080000:3.0:1734018440.400820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018440.400823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018440.400827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018440.400836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334459904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018440.400842:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334459520:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018440.400845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018440.400848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018440.400850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018440.400853:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2707 00000100:00080000:3.0:1734018440.400855:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018440.400859:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018440.400863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018440.400871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334460032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018440.400879:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018440.400881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018440.400883:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018440.400886:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2707 00000100:00080000:3.0:1734018440.400889:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018440.400892:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018440.400896:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018440.400903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334460160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018440.400908:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018440.400912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018440.400915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018440.400917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018440.400920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2707 00000100:00080000:3.0:1734018440.400925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018440.400929:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018440.400930:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018440/real 1734018440] req@ffff8800a7980a80 x1818247334459520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018440.400933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018440.400940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334460288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018440.400958:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334459520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018440.400969:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018440.400972:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018440.400977:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334459520:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018440.400985:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334459648:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018440.401001:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018440.401009:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018440/real 1734018440] req@ffff8800a7981f80 x1818247334459648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018440.401020:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334459648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018440.401062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018440.401065:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018440.401069:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334459648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018440.401076:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334459776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018440.401094:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018440.401101:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018440/real 1734018440] req@ffff8800a7982300 x1818247334459776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018440.401112:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334459776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018440.401120:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018440.401123:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018440.401126:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334459776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018440.401154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334459904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018440.401168:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018440.401175:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018440/real 1734018440] req@ffff8800a7980000 x1818247334459904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018440.401185:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334459904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018440.401192:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018440.401194:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018440.401197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334459904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018440.401202:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334460032:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018440.401215:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018440.401221:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018440/real 1734018440] req@ffff8800a7983480 x1818247334460032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018440.401231:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334460032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018440.401238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018440.401240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018440.401243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334460032:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018440.401249:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334460160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018440.401264:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018440.401271:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018440/real 1734018440] req@ffff8800a7982680 x1818247334460160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018440.401281:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334460160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018440.401288:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018440.401290:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018440.401293:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334460160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018440.401298:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334460288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018440.401310:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018440.401316:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018440/real 1734018440] req@ffff8800a7981880 x1818247334460288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018440.401326:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334460288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018440.401333:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018440.401335:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018440.401338:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334460288:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018441.224742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018442.226790:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018443.228775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018444.230778:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018445.232759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018445.408609:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018445.408618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018445.408622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018445.408633:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2712 00000100:00080000:3.0:1734018445.408642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018445.408650:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018445.408686:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981c00 x1818247334460416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018445.408723:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018445.408727:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018445.408730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018445.408735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2712 00000100:00080000:3.0:1734018445.408738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018445.408742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018445.408752:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018445.408764:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980380 x1818247334460544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018445.408774:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018445.408777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018445.408779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018445.408783:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2712 00000100:00080000:3.0:1734018445.408786:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018445.408790:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018445.408794:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018445.408804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981880 x1818247334460672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018445.408812:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018445.408815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018445.408817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018445.408821:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2712 00000100:00080000:3.0:1734018445.408823:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018445.408826:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018445.408830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018445.408839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982680 x1818247334460800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018445.408840:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334460416:192.168.201.116@tcp:250:kworker.0 00000100:00080000:3.0:1734018445.408848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018445.408852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018445.408853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018445.408856:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2712 00000100:00080000:3.0:1734018445.408859:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018445.408862:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018445.408866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018445.408873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983480 x1818247334460928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018445.408881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018445.408884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018445.408886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018445.408889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2712 00000100:00080000:3.0:1734018445.408892:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018445.408895:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018445.408899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018445.408907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980000 x1818247334461056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018445.408914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018445.408917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018445.408919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018445.408922:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2712 00000100:00080000:3.0:1734018445.408925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018445.408929:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018445.408933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018445.408939:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334460544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:3.0:1734018445.408981:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982300 x1818247334461184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018445.409011:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334460672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018445.409046:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334460800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018445.409070:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334460928:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018445.409091:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334461056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018445.409114:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334461184:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018446.234751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018447.236752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018448.238732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734018448.414754:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:2.0:1734018448.414803:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734018448.414821:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018448.414859:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018448.414866:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018448.414873:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018448.414879:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018448.414885:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:2.0:1734018448.414917:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:0.0:1734018448.414985:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018445/real 1734018448] req@ffff8800a7982680 x1818247334460800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018448.415024:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982680 x1818247334460800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018448.415040:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018448.415046:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018448.415054:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334460800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018448.415066:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018445/real 1734018448] req@ffff8800a7983480 x1818247334460928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018448.415086:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983480 x1818247334460928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018448.415099:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018448.415103:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018448.415109:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334460928:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018448.415119:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018445/real 1734018448] req@ffff8800a7980000 x1818247334461056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018448.415136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980000 x1818247334461056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018448.415147:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018448.415152:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018448.415157:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334461056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018448.415167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018445/real 1734018448] req@ffff8800a7982300 x1818247334461184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018448.415179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982300 x1818247334461184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018448.415187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018448.415190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018448.415193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334461184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018448.415199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018445/real 1734018448] req@ffff8800a7981c00 x1818247334460416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018448.415213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981c00 x1818247334460416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018448.415225:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018448.415228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018448.415233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334460416:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018448.415243:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018445/real 1734018448] req@ffff8800a7980380 x1818247334460544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018448.415260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980380 x1818247334460544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018448.415285:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018448.415289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018448.415296:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334460544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018448.415306:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018445/real 1734018448] req@ffff8800a7981880 x1818247334460672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018448.415324:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981880 x1818247334460672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018448.415336:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018448.415340:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018448.415345:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334460672:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018449.240771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018450.242749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018450.416625:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018450.416634:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018450.416639:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018450.416649:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2717 00000100:00080000:3.0:1734018450.416658:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018450.416665:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018450.416703:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981f80 x1818247334461312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018450.416739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018450.416743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018450.416745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018450.416749:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2717 00000100:00080000:3.0:1734018450.416752:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018450.416756:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018450.416767:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018450.416779:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980a80 x1818247334461440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018450.416797:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018450.416800:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018450.416802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018450.416806:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2717 00000100:00080000:3.0:1734018450.416809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018450.416813:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018450.416818:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018450.416827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334461568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018450.416835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018450.416839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018450.416841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018450.416844:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2717 00000100:00080000:3.0:1734018450.416846:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018450.416850:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734018450.416853:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334461312:192.168.201.116@tcp:250:kworker.0 00000008:00100000:3.0:1734018450.416853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018450.416895:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334461696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018450.416905:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018450.416908:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018450.416910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018450.416914:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2717 00000100:00080000:3.0:1734018450.416917:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018450.416920:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018450.416925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018450.416932:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334461824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018450.416941:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018450.416944:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018450.416946:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018450.416949:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2717 00000100:00080000:3.0:1734018450.416952:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734018450.416956:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018450.416956:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018450.416959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018450.416967:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334461952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018450.416974:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018450.416978:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018450.416979:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018450.416981:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018450/real 1734018450] req@ffff8800a7981f80 x1818247334461312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018450.416983:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2717 00000100:00080000:3.0:1734018450.416986:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018450.416989:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018450.416992:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018450.417000:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334462080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018450.417001:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981f80 x1818247334461312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018450.417011:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018450.417015:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018450.417019:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334461312:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018450.417027:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334461440:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018450.417043:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018450.417051:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018450/real 1734018450] req@ffff8800a7980a80 x1818247334461440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018450.417063:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980a80 x1818247334461440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018450.417072:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018450.417075:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018450.417079:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334461440:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018450.417085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334461568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018450.417099:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018450.417105:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018450/real 1734018450] req@ffff8800a7982d80 x1818247334461568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018450.417115:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334461568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018450.417123:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018450.417125:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018450.417129:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334461568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018450.417153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334461696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018450.417166:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018450.417172:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018450/real 1734018450] req@ffff8800a7983800 x1818247334461696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018450.417183:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334461696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018450.417191:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018450.417193:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018450.417195:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334461696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018450.417202:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334461824:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018450.417215:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018450.417221:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018450/real 1734018450] req@ffff8800a7980e00 x1818247334461824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018450.417231:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334461824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018450.417239:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018450.417241:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018450.417245:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334461824:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018450.417250:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334461952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018450.417261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018450.417267:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018450/real 1734018450] req@ffff8800a7981500 x1818247334461952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018450.417277:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334461952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018450.417284:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018450.417287:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018450.417290:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334461952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018450.417296:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334462080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018450.417308:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018450.417314:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018450/real 1734018450] req@ffff8800a7983b80 x1818247334462080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018450.417324:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334462080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018450.417332:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018450.417334:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018450.417337:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334462080:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018451.244768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018452.246770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018453.248775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018454.250768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018455.252769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018455.424578:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018455.424587:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018455.424591:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018455.424602:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2722 00000100:00080000:3.0:1734018455.424611:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018455.424619:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:3.0:1734018455.424656:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980700 x1818247334462208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018455.424690:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018455.424694:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018455.424696:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018455.424700:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2722 00000100:00080000:3.0:1734018455.424704:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018455.424708:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018455.424718:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018455.424729:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981180 x1818247334462336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018455.424739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018455.424742:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018455.424745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018455.424749:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2722 00000100:00080000:3.0:1734018455.424752:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018455.424756:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018455.424761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1734018455.424800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983b80 x1818247334462464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018455.424810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018455.424813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018455.424815:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018455.424819:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2722 00000100:00080000:3.0:1734018455.424822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018455.424825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734018455.424827:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334462208:192.168.201.116@tcp:250:kworker.0 00000008:00100000:3.0:1734018455.424830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:3.0:1734018455.424838:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7981500 x1818247334462592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018455.424848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018455.424851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018455.424853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018455.424856:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2722 00000100:00080000:3.0:1734018455.424859:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018455.424862:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018455.424866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018455.424873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7980e00 x1818247334462720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018455.424882:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018455.424885:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018455.424887:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018455.424890:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2722 00000800:00000100:0.0:1734018455.424893:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:3.0:1734018455.424893:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018455.424896:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018455.424900:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018455.424908:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7983800 x1818247334462848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018455.424916:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734018455.424918:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018455/real 1734018455] req@ffff8800a7980700 x1818247334462208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018455.424918:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018455.424920:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018455.424923:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2722 00000100:00080000:3.0:1734018455.424926:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018455.424929:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:3.0:1734018455.424933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018455.424939:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980700 x1818247334462208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018455.424940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a7982d80 x1818247334462976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018455.424949:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018455.424953:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018455.424958:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334462208:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018455.424965:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334462336:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018455.424982:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018455.424989:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018455/real 1734018455] req@ffff8800a7981180 x1818247334462336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018455.425000:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981180 x1818247334462336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018455.425008:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018455.425012:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018455.425015:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334462336:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018455.425022:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334462464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018455.425035:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018455.425041:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018455/real 1734018455] req@ffff8800a7983b80 x1818247334462464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018455.425051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983b80 x1818247334462464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018455.425059:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018455.425061:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018455.425065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334462464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018455.425089:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334462592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018455.425102:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018455.425108:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018455/real 1734018455] req@ffff8800a7981500 x1818247334462592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018455.425118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7981500 x1818247334462592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018455.425127:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018455.425129:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018455.425132:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7981500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334462592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018455.425137:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334462720:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018455.425152:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018455.425159:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018455/real 1734018455] req@ffff8800a7980e00 x1818247334462720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018455.425169:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7980e00 x1818247334462720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018455.425176:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018455.425178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018455.425181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7980e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334462720:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018455.425186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334462848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018455.425198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018455.425205:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018455/real 1734018455] req@ffff8800a7983800 x1818247334462848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018455.425214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7983800 x1818247334462848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018455.425222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018455.425224:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018455.425227:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7983800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334462848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018455.425232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334462976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018455.425244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018455.425251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018455/real 1734018455] req@ffff8800a7982d80 x1818247334462976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018455.425267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a7982d80 x1818247334462976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018455.425275:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018455.425277:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018455.425280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a7982d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334462976:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018456.254815:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018457.256735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018458.258800:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018459.260799:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018460.262765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018460.432744:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018460.432756:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018460.432762:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018460.432775:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2727 00000100:00080000:0.0:1734018460.432787:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018460.432796:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018460.432844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334463104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018460.432885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018460.432890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018460.432892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018460.432897:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2727 00000100:00080000:0.0:1734018460.432900:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018460.432904:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018460.432916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018460.432928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334463232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018460.432946:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018460.432949:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018460.432951:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018460.432955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2727 00000100:00080000:0.0:1734018460.432958:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018460.432961:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018460.432966:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018460.432975:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334463360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018460.432984:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018460.432987:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018460.432989:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018460.432992:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2727 00000100:00100000:2.0:1734018460.432993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334463104:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018460.432995:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018460.432998:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018460.433003:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018460.433012:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334463488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018460.433021:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018460.433024:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018460.433025:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018460.433029:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2727 00000100:00080000:0.0:1734018460.433033:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018460.433036:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018460.433040:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018460.433049:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334463616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018460.433057:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018460.433060:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018460.433062:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018460.433065:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2727 00000100:00080000:0.0:1734018460.433068:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018460.433071:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018460.433076:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018460.433078:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018460.433084:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334463744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018460.433092:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018460.433095:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018460.433096:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018460.433100:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2727 00000100:00080000:0.0:1734018460.433103:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018460.433105:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018460/real 1734018460] req@ffff8800a934df80 x1818247334463104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018460.433106:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018460.433111:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018460.433118:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334463872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018460.433127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334463104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018460.433139:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018460.433143:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018460.433148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334463104:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018460.433156:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334463232:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018460.433174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018460.433182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018460/real 1734018460] req@ffff8800a934d880 x1818247334463232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018460.433194:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334463232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018460.433201:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018460.433204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018460.433207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334463232:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018460.433234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334463360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018460.433248:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018460.433255:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018460/real 1734018460] req@ffff8800a934dc00 x1818247334463360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018460.433265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334463360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018460.433273:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018460.433275:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018460.433279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334463360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018460.433285:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334463488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018460.433297:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018460.433303:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018460/real 1734018460] req@ffff8800a934e300 x1818247334463488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018460.433313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334463488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018460.433328:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018460.433330:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018460.433333:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334463488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018460.433339:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334463616:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018460.433353:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018460.433359:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018460/real 1734018460] req@ffff8800a934f100 x1818247334463616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018460.433369:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334463616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018460.433376:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018460.433379:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018460.433382:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334463616:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018460.433388:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334463744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018460.433401:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018460.433407:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018460/real 1734018460] req@ffff8800a934fb80 x1818247334463744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018460.433417:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334463744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018460.433424:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018460.433426:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018460.433429:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334463744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018460.433435:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334463872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018460.433448:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018460.433454:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018460/real 1734018460] req@ffff8800a934d500 x1818247334463872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018460.433505:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334463872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018460.433512:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018460.433514:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018460.433518:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334463872:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018461.264761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018462.266744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018463.268738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018464.270761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018465.272738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018465.440642:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018465.440656:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018465.440661:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018465.440671:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2732 00000100:00080000:0.0:1734018465.440680:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018465.440688:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018465.440726:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334464000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018465.440760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018465.440764:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018465.440766:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018465.440771:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2732 00000100:00080000:0.0:1734018465.440774:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018465.440778:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018465.440788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018465.440800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334464128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018465.440809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018465.440812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018465.440814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018465.440818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2732 00000100:00080000:0.0:1734018465.440821:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018465.440825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018465.440830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018465.440841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334464256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018465.440850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018465.440853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018465.440854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018465.440858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2732 00000100:00080000:0.0:1734018465.440860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018465.440863:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018465.440868:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018465.440868:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334464000:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018465.440875:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334464384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018465.440885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018465.440888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018465.440890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018465.440893:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2732 00000100:00080000:0.0:1734018465.440895:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018465.440899:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018465.440902:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018465.440910:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334464512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018465.440917:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018465.440920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018465.440922:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018465.440926:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2732 00000100:00080000:0.0:1734018465.440929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018465.440932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734018465.440935:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018465.440936:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018465.440944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334464640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018465.440952:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018465.440955:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018465.440956:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018465.440958:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018465/real 1734018465] req@ffff8800a934ce00 x1818247334464000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018465.440960:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2732 00000100:00080000:0.0:1734018465.440962:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018465.440965:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018465.440970:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018465.440977:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334464768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018465.440977:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334464000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018465.440988:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018465.440991:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018465.440996:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334464000:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018465.441003:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334464128:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018465.441019:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018465.441027:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018465/real 1734018465] req@ffff8800a934e680 x1818247334464128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018465.441046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334464128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018465.441054:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018465.441057:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018465.441060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334464128:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018465.441066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334464256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018465.441079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018465.441085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018465/real 1734018465] req@ffff8800a934d500 x1818247334464256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018465.441095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334464256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018465.441103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018465.441105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018465.441108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334464256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018465.441132:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334464384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018465.441145:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018465.441152:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018465/real 1734018465] req@ffff8800a934fb80 x1818247334464384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018465.441162:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334464384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018465.441169:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018465.441172:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018465.441175:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334464384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018465.441181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334464512:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018465.441193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018465.441199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018465/real 1734018465] req@ffff8800a934f100 x1818247334464512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018465.441209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334464512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018465.441216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018465.441218:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018465.441222:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334464512:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018465.441228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334464640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018465.441244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018465.441250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018465/real 1734018465] req@ffff8800a934e300 x1818247334464640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018465.441260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334464640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018465.441267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018465.441269:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018465.441272:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334464640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018465.441277:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334464768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018465.441292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018465.441298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018465/real 1734018465] req@ffff8800a934dc00 x1818247334464768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018465.441308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334464768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018465.441315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018465.441317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018465.441320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334464768:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018466.274745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018467.276812:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018468.278753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018469.280793:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018470.282762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018470.448631:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018470.448645:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018470.448649:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018470.448660:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2737 00000100:00080000:0.0:1734018470.448670:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018470.448677:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018470.448714:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334464896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018470.448755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018470.448759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018470.448761:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018470.448766:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2737 00000100:00080000:0.0:1734018470.448769:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018470.448773:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018470.448783:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018470.448794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334465024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018470.448803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018470.448806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018470.448808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018470.448812:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2737 00000100:00080000:0.0:1734018470.448814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018470.448818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018470.448823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018470.448831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334465152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018470.448840:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018470.448842:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018470.448844:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018470.448847:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2737 00000100:00080000:0.0:1734018470.448850:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018470.448854:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018470.448858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018470.448862:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334464896:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018470.448866:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334465280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018470.448874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018470.448877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018470.448879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018470.448882:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2737 00000100:00080000:0.0:1734018470.448884:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018470.448887:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018470.448891:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018470.448899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334465408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018470.448906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018470.448909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018470.448911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018470.448914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2737 00000100:00080000:0.0:1734018470.448917:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018470.448920:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018470.448924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018470.448929:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018470.448934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334465536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018470.448941:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018470.448945:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018470.448947:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018470.448951:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2737 00000100:00100000:2.0:1734018470.448953:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018470/real 1734018470] req@ffff8800a934d880 x1818247334464896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018470.448954:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018470.448957:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018470.448961:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018470.448969:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334465664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018470.448974:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334464896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018470.448984:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018470.448987:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018470.448992:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334464896:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018470.448999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334465024:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018470.449015:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018470.449021:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018470/real 1734018470] req@ffff8800a934df80 x1818247334465024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018470.449032:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334465024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018470.449041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018470.449043:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018470.449047:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334465024:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018470.449052:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334465152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018470.449066:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018470.449072:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018470/real 1734018470] req@ffff8800a934c700 x1818247334465152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018470.449082:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334465152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018470.449089:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018470.449091:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018470.449095:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334465152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018470.449124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334465280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018470.449138:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018470.449144:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018470/real 1734018470] req@ffff8800a934c000 x1818247334465280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018470.449154:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334465280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018470.449162:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018470.449164:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018470.449167:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334465280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018470.449173:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334465408:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018470.449185:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018470.449191:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018470/real 1734018470] req@ffff8800a934d180 x1818247334465408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018470.449201:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334465408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018470.449208:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018470.449210:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018470.449213:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334465408:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018470.449219:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334465536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018470.449230:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018470.449236:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018470/real 1734018470] req@ffff8800a934ea00 x1818247334465536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018470.449246:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334465536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018470.449253:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018470.449256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018470.449259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334465536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018470.449264:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334465664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018470.449276:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018470.449282:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018470/real 1734018470] req@ffff8800a934c380 x1818247334465664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018470.449293:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334465664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018470.449301:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018470.449303:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018470.449306:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334465664:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018471.284768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018472.286837:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018473.288752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018474.290696:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018475.292718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018475.456587:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018475.456597:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018475.456601:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018475.456612:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2742 00000100:00080000:0.0:1734018475.456622:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018475.456630:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018475.456667:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334465792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018475.456701:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018475.456705:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018475.456708:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018475.456712:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2742 00000100:00080000:0.0:1734018475.456716:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018475.456720:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018475.456730:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018475.456741:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334465920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018475.456752:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018475.456755:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018475.456757:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018475.456761:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2742 00000100:00080000:0.0:1734018475.456764:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018475.456767:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018475.456772:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018475.456782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334466048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018475.456791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018475.456794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018475.456796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018475.456799:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2742 00000100:00080000:0.0:1734018475.456801:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018475.456805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018475.456809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018475.456810:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334465792:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018475.456817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334466176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018475.456827:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018475.456830:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018475.456831:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018475.456835:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2742 00000100:00080000:0.0:1734018475.456838:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018475.456841:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018475.456845:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018475.456852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334466304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018475.456860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018475.456863:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018475.456865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018475.456868:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2742 00000100:00080000:0.0:1734018475.456871:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018475.456874:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734018475.456876:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018475.456878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018475.456886:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334466432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018475.456894:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018475.456897:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018475.456899:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018475.456899:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018475/real 1734018475] req@ffff8800a934ed80 x1818247334465792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018475.456902:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2742 00000100:00080000:0.0:1734018475.456904:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018475.456907:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018475.456911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018475.456918:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334465792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018475.456919:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334466560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018475.456929:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018475.456932:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018475.456937:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334465792:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018475.456945:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334465920:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018475.456962:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018475.456969:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018475/real 1734018475] req@ffff8800a934ca80 x1818247334465920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018475.456980:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334465920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018475.456988:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018475.456991:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018475.456995:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334465920:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018475.457001:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334466048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018475.457015:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018475.457021:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018475/real 1734018475] req@ffff8800a934c380 x1818247334466048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018475.457031:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334466048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018475.457039:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018475.457042:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018475.457045:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334466048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018475.457069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334466176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018475.457081:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018475.457088:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018475/real 1734018475] req@ffff8800a934ea00 x1818247334466176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018475.457098:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334466176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018475.457106:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018475.457108:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018475.457111:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334466176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018475.457116:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334466304:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018475.457131:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018475.457137:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018475/real 1734018475] req@ffff8800a934d180 x1818247334466304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018475.457148:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334466304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018475.457155:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018475.457157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018475.457160:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334466304:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018475.457165:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334466432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018475.457176:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018475.457182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018475/real 1734018475] req@ffff8800a934c000 x1818247334466432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018475.457192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334466432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018475.457199:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018475.457201:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018475.457204:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334466432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018475.457209:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334466560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018475.457221:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018475.457227:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018475/real 1734018475] req@ffff8800a934c700 x1818247334466560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018475.457243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334466560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018475.457250:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018475.457252:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018475.457255:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334466560:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018476.294754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018477.296728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018478.298777:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018479.300692:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018480.302746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018480.464600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018480.464617:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018480.464624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018480.464642:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2747 00000100:00080000:0.0:1734018480.464657:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018480.464670:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018480.464727:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334466688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018480.464768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018480.464773:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018480.464775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018480.464780:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2747 00000100:00080000:0.0:1734018480.464784:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018480.464789:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018480.464800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018480.464814:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334466816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018480.464824:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018480.464827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018480.464829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018480.464833:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2747 00000100:00080000:0.0:1734018480.464836:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018480.464840:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018480.464846:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018480.464855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334466944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018480.464865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018480.464868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018480.464870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018480.464873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2747 00000100:00080000:0.0:1734018480.464876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018480.464879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018480.464884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018480.464888:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334466688:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018480.464892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334467072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018480.464902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018480.464905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018480.464907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018480.464910:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2747 00000100:00080000:0.0:1734018480.464913:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018480.464916:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018480.464920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018480.464929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334467200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018480.464938:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018480.464941:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018480.464943:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018480.464947:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2747 00000100:00080000:0.0:1734018480.464951:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018480.464954:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018480.464958:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018480.464961:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018480.464969:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334467328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018480.464977:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018480.464981:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018480.464982:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018480.464986:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2747 00000100:00100000:2.0:1734018480.464986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018480/real 1734018480] req@ffff8800a934df80 x1818247334466688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018480.464989:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018480.464992:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018480.464997:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018480.465005:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334467456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018480.465006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334466688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018480.465018:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018480.465022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018480.465027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334466688:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018480.465035:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334466816:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018480.465053:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018480.465060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018480/real 1734018480] req@ffff8800a934d880 x1818247334466816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018480.465071:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334466816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018480.465080:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018480.465083:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018480.465087:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334466816:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018480.465094:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334466944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018480.465108:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018480.465114:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018480/real 1734018480] req@ffff8800a934dc00 x1818247334466944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018480.465124:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334466944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018480.465133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018480.465135:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018480.465139:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334466944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018480.465164:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334467072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018480.465177:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018480.465184:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018480/real 1734018480] req@ffff8800a934e300 x1818247334467072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018480.465194:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334467072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018480.465202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018480.465205:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018480.465208:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334467072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018480.465214:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334467200:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018480.465230:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018480.465236:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018480/real 1734018480] req@ffff8800a934f100 x1818247334467200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018480.465246:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334467200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018480.465254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018480.465256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018480.465259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334467200:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018480.465265:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334467328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018480.465277:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018480.465283:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018480/real 1734018480] req@ffff8800a934fb80 x1818247334467328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018480.465292:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334467328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018480.465299:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018480.465302:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018480.465305:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334467328:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018480.465310:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334467456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018480.465323:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018480.465329:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018480/real 1734018480] req@ffff8800a934d500 x1818247334467456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018480.465386:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334467456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018480.465396:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018480.465398:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018480.465401:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334467456:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018481.304730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018482.306700:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018483.308732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018484.310730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018485.312756:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018485.472603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018485.472616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018485.472621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018485.472636:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2752 00000100:00080000:0.0:1734018485.472648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018485.472658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018485.472707:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334467584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018485.472755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018485.472759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018485.472761:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018485.472766:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2752 00000100:00080000:0.0:1734018485.472769:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018485.472773:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018485.472785:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018485.472797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334467712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018485.472806:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018485.472810:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018485.472811:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018485.472816:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2752 00000100:00080000:0.0:1734018485.472819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018485.472823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018485.472828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018485.472840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334467840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018485.472849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018485.472853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018485.472854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018485.472858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2752 00000100:00080000:0.0:1734018485.472861:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018485.472865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018485.472870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018485.472879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334467968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018485.472889:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734018485.472889:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334467584:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018485.472892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018485.472893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018485.472897:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2752 00000100:00080000:0.0:1734018485.472900:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018485.472903:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018485.472907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018485.472916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334468096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018485.472924:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018485.472927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018485.472928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018485.472932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2752 00000100:00080000:0.0:1734018485.472935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018485.472938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018485.472942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018485.472951:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334468224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018485.472959:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1734018485.472962:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018485.472963:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018485.472965:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018485.472969:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2752 00000100:00080000:0.0:1734018485.472972:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018485.472976:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018485.472980:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018485.472988:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334468352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018485.472989:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018485/real 1734018485] req@ffff8800a934e680 x1818247334467584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018485.473010:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334467584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018485.473023:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018485.473026:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018485.473031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334467584:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018485.473039:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334467712:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018485.473057:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018485.473064:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018485/real 1734018485] req@ffff8800a934ce00 x1818247334467712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018485.473075:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334467712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018485.473083:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018485.473087:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018485.473090:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334467712:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018485.473097:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334467840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018485.473110:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018485.473116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018485/real 1734018485] req@ffff8800a934d500 x1818247334467840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018485.473127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334467840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018485.473134:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018485.473137:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018485.473141:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334467840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018485.473167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334467968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018485.473180:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018485.473187:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018485/real 1734018485] req@ffff8800a934fb80 x1818247334467968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018485.473198:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334467968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018485.473206:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018485.473208:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018485.473211:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334467968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018485.473218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334468096:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018485.473234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018485.473240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018485/real 1734018485] req@ffff8800a934f100 x1818247334468096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018485.473251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334468096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018485.473258:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018485.473303:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018485.473308:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334468096:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018485.473314:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334468224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018485.473331:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018485.473339:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018485/real 1734018485] req@ffff8800a934e300 x1818247334468224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018485.473349:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334468224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018485.473357:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018485.473360:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018485.473363:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334468224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018485.473369:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334468352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018485.473382:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018485.473388:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018485/real 1734018485] req@ffff8800a934dc00 x1818247334468352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018485.473405:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334468352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018485.473412:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018485.473415:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018485.473418:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334468352:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018486.314764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018487.316741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018488.318768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018489.320756:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018490.322757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018490.480603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018490.480612:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018490.480617:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018490.480628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2757 00000100:00080000:0.0:1734018490.480637:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018490.480644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018490.480681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334468480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018490.480716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018490.480720:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018490.480722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018490.480727:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2757 00000100:00080000:0.0:1734018490.480730:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018490.480734:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018490.480744:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018490.480755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334468608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018490.480766:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018490.480770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018490.480771:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018490.480775:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2757 00000100:00080000:0.0:1734018490.480778:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018490.480781:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018490.480786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018490.480794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334468736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018490.480803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018490.480806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018490.480807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018490.480811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2757 00000100:00080000:0.0:1734018490.480814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018490.480817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018490.480820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018490.480828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334468864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018490.480829:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334468480:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018490.480838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018490.480840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018490.480842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018490.480845:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2757 00000100:00080000:0.0:1734018490.480849:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018490.480852:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018490.480856:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018490.480863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334468992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018490.480871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018490.480874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018490.480876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018490.480879:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2757 00000100:00080000:0.0:1734018490.480882:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018490.480885:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018490.480889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018490.480895:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018490.480898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334469120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018490.480906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018490.480908:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018490.480910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018490.480913:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2757 00000100:00080000:0.0:1734018490.480916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018490.480917:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018490/real 1734018490] req@ffff8800a934d880 x1818247334468480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018490.480919:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018490.480923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018490.480930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334469248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018490.480937:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334468480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018490.480948:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018490.480952:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018490.480956:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334468480:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018490.480963:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334468608:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018490.480979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018490.480987:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018490/real 1734018490] req@ffff8800a934df80 x1818247334468608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018490.480998:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334468608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018490.481005:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018490.481008:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018490.481012:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334468608:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018490.481018:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334468736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018490.481031:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018490.481038:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018490/real 1734018490] req@ffff8800a934c700 x1818247334468736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018490.481048:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334468736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018490.481055:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018490.481058:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018490.481061:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334468736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018490.481085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334468864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018490.481098:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018490.481104:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018490/real 1734018490] req@ffff8800a934c000 x1818247334468864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018490.481115:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334468864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018490.481122:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018490.481125:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018490.481128:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334468864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018490.481133:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334468992:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018490.481148:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018490.481154:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018490/real 1734018490] req@ffff8800a934d180 x1818247334468992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018490.481165:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334468992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018490.481172:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018490.481174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018490.481221:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334468992:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018490.481228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334469120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018490.481244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018490.481251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018490/real 1734018490] req@ffff8800a934ea00 x1818247334469120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018490.481261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334469120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018490.481269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018490.481271:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018490.481274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334469120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018490.481280:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334469248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018490.481292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018490.481298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018490/real 1734018490] req@ffff8800a934c380 x1818247334469248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018490.481315:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334469248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018490.481323:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018490.481325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018490.481328:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334469248:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018491.324706:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018492.326791:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018493.328783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018494.330863:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018495.332708:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018495.488603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018495.488612:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018495.488617:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018495.488628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2762 00000100:00080000:0.0:1734018495.488638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018495.488647:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018495.488684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334469376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018495.488721:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018495.488725:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018495.488727:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018495.488732:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2762 00000100:00080000:0.0:1734018495.488735:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018495.488739:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018495.488749:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018495.488759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334469504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018495.488769:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018495.488773:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018495.488774:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018495.488779:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2762 00000100:00080000:0.0:1734018495.488781:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018495.488785:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018495.488791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018495.488802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334469632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018495.488812:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018495.488815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018495.488816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018495.488820:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2762 00000100:00080000:0.0:1734018495.488822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018495.488825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018495.488829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018495.488837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334469760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018495.488847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018495.488850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018495.488852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018495.488855:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2762 00000100:00080000:0.0:1734018495.488857:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018495.488861:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018495.488865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018495.488872:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334469376:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018495.488874:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334469888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018495.488881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018495.488884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018495.488887:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018495.488891:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2762 00000100:00080000:0.0:1734018495.488893:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018495.488897:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018495.488901:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018495.488909:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334470016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018495.488917:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018495.488920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018495.488921:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018495.488924:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2762 00000100:00080000:0.0:1734018495.488927:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018495.488930:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018495.488933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018495.488940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334470144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734018495.488972:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018495.489003:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018495/real 1734018495] req@ffff8800a934ca80 x1818247334469376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018495.489025:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334469376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018495.489039:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018495.489043:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018495.489048:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334469376:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018495.489056:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334469504:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018495.489075:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018495.489082:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018495/real 1734018495] req@ffff8800a934ed80 x1818247334469504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018495.489144:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334469504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018495.489153:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018495.489157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018495.489161:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334469504:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018495.489169:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334469632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018495.489188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018495.489195:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018495/real 1734018495] req@ffff8800a934c380 x1818247334469632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018495.489207:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334469632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018495.489214:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018495.489217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018495.489220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334469632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018495.489251:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334469760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018495.489265:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018495.489272:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018495/real 1734018495] req@ffff8800a934ea00 x1818247334469760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018495.489283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334469760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018495.489291:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018495.489294:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018495.489297:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334469760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018495.489303:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334469888:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018495.489319:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018495.489326:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018495/real 1734018495] req@ffff8800a934d180 x1818247334469888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018495.489337:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334469888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018495.489344:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018495.489346:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018495.489349:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334469888:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018495.489355:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334470016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018495.489368:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018495.489374:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018495/real 1734018495] req@ffff8800a934c000 x1818247334470016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018495.489384:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334470016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018495.489391:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018495.489393:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018495.489396:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334470016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018495.489402:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334470144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018495.489416:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018495.489422:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018495/real 1734018495] req@ffff8800a934c700 x1818247334470144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018495.489438:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334470144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018495.489446:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018495.489448:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018495.489452:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334470144:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018496.334720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018497.336917:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018498.338633:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018499.340606:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018500.342639:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018500.496568:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018500.496577:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018500.496581:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018500.496592:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2767 00000100:00080000:0.0:1734018500.496601:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018500.496608:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018500.496645:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334470272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018500.496687:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018500.496691:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018500.496693:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018500.496697:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2767 00000100:00080000:0.0:1734018500.496700:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018500.496704:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018500.496714:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018500.496725:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334470400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018500.496735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018500.496738:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018500.496740:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018500.496744:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2767 00000100:00080000:0.0:1734018500.496747:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018500.496751:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018500.496755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018500.496763:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334470528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018500.496772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018500.496775:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018500.496776:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018500.496780:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2767 00000100:00080000:0.0:1734018500.496782:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018500.496786:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018500.496790:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018500.496795:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334470272:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018500.496797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334470656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018500.496806:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018500.496809:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018500.496811:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018500.496814:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2767 00000100:00080000:0.0:1734018500.496816:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018500.496820:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018500.496823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018500.496830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334470784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018500.496838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018500.496841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018500.496842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018500.496846:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2767 00000100:00080000:0.0:1734018500.496849:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018500.496852:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734018500.496862:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018500.496867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018500.496879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334470912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018500.496885:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018500/real 1734018500] req@ffff8800a934df80 x1818247334470272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018500.496887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018500.496891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018500.496893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018500.496897:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2767 00000100:00080000:0.0:1734018500.496900:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018500.496903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018500.496905:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334470272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018500.496907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018500.496915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334471040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018500.496916:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018500.496919:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018500.496924:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334470272:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018500.496932:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334470400:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018500.496948:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018500.496955:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018500/real 1734018500] req@ffff8800a934d880 x1818247334470400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018500.496967:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334470400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018500.496974:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018500.496978:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018500.496981:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334470400:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018500.496987:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334470528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018500.497001:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018500.497046:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018500/real 1734018500] req@ffff8800a934dc00 x1818247334470528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018500.497058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334470528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018500.497066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018500.497068:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018500.497072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334470528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018500.497097:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334470656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018500.497114:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018500.497121:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018500/real 1734018500] req@ffff8800a934e300 x1818247334470656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018500.497131:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334470656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018500.497139:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018500.497141:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018500.497144:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334470656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018500.497149:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334470784:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018500.497165:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018500.497171:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018500/real 1734018500] req@ffff8800a934f100 x1818247334470784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018500.497182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334470784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018500.497189:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018500.497191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018500.497194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334470784:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018500.497199:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334470912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018500.497211:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018500.497217:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018500/real 1734018500] req@ffff8800a934fb80 x1818247334470912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018500.497227:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334470912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018500.497234:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018500.497236:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018500.497239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334470912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018500.497244:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334471040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018500.497257:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018500.497263:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018500/real 1734018500] req@ffff8800a934d500 x1818247334471040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018500.497279:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334471040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018500.497287:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018500.497289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018500.497292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334471040:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018501.344598:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018502.346632:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018503.348744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018504.350796:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018505.352718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018505.504598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018505.504608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018505.504613:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018505.504624:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2772 00000100:00080000:0.0:1734018505.504635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018505.504642:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018505.504680:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334471168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018505.504716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018505.504720:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018505.504722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018505.504726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2772 00000100:00080000:0.0:1734018505.504729:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018505.504733:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018505.504743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018505.504754:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334471296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018505.504765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018505.504768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018505.504770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018505.504774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2772 00000100:00080000:0.0:1734018505.504776:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018505.504780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018505.504784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018505.504794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334471424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018505.504803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018505.504806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018505.504808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018505.504811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2772 00000100:00080000:0.0:1734018505.504814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018505.504817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018505.504821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018505.504829:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334471552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018505.504838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018505.504841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018505.504843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018505.504846:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2772 00000100:00080000:0.0:1734018505.504850:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018505.504850:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334471168:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018505.504853:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018505.504858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018505.504865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334471680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018505.504873:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018505.504876:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018505.504878:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018505.504881:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2772 00000100:00080000:0.0:1734018505.504883:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018505.504886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018505.504890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018505.504898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334471808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018505.504906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018505.504909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018505.504911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018505.504914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2772 00000100:00080000:0.0:1734018505.504918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018505.504950:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018505.504955:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018505.504963:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334471936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734018505.504987:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018505.505019:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018505/real 1734018505] req@ffff8800a934ce00 x1818247334471168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018505.505042:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334471168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018505.505054:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018505.505058:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018505.505064:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334471168:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018505.505072:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334471296:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018505.505093:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018505.505100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018505/real 1734018505] req@ffff8800a934e680 x1818247334471296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018505.505111:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334471296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018505.505119:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018505.505124:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018505.505127:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334471296:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018505.505134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334471424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018505.505149:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018505.505156:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018505/real 1734018505] req@ffff8800a934d500 x1818247334471424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018505.505166:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334471424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018505.505174:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018505.505177:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018505.505181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334471424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018505.505212:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334471552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018505.505225:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018505.505232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018505/real 1734018505] req@ffff8800a934fb80 x1818247334471552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018505.505242:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334471552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018505.505251:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018505.505253:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018505.505256:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334471552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018505.505263:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334471680:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018505.505279:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018505.505286:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018505/real 1734018505] req@ffff8800a934f100 x1818247334471680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018505.505297:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334471680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018505.505304:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018505.505306:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018505.505309:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334471680:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018505.505315:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334471808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018505.505328:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018505.505334:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018505/real 1734018505] req@ffff8800a934e300 x1818247334471808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018505.505344:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334471808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018505.505353:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018505.505355:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018505.505358:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334471808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018505.505364:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334471936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018505.505378:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018505.505384:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018505/real 1734018505] req@ffff8800a934dc00 x1818247334471936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018505.505400:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334471936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018505.505408:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018505.505411:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018505.505414:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334471936:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018506.354757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018507.356750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018508.358700:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018509.360748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018510.362731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018510.512584:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018510.512593:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018510.512597:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018510.512608:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2777 00000100:00080000:0.0:1734018510.512616:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018510.512624:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018510.512661:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334472064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018510.512695:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018510.512699:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018510.512701:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018510.512705:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2777 00000100:00080000:0.0:1734018510.512708:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018510.512712:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018510.512722:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018510.512733:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334472192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018510.512743:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018510.512746:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018510.512748:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018510.512752:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2777 00000100:00080000:0.0:1734018510.512755:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018510.512758:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018510.512764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018510.512773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334472320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018510.512781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018510.512785:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018510.512786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018510.512789:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2777 00000100:00080000:0.0:1734018510.512792:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018510.512795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018510.512799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018510.512801:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334472064:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018510.512807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334472448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018510.512816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018510.512819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018510.512821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018510.512824:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2777 00000100:00080000:0.0:1734018510.512826:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018510.512829:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018510.512833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018510.512875:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334472576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018510.512884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018510.512887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018510.512889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018510.512893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2777 00000100:00080000:0.0:1734018510.512896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018510.512899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018510.512904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018510.512912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334472704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018510.512920:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018510.512923:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018510.512925:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018510.512928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2777 00000100:00100000:2.0:1734018510.512929:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334472192:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734018510.512931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018510.512934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018510.512938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018510.512945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334472832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018510.512950:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334472320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018510.512973:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334472448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018510.512988:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334472576:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018510.513002:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334472704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018510.513016:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334472832:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018511.364767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018512.366716:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018513.368836:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018513.518829:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734018513.518881:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018513.518901:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018513.518946:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018513.518956:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018513.518963:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018513.518970:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018513.518977:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018513.518984:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:2.0:1734018513.519066:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018510/real 1734018513] req@ffff8800a934c000 x1818247334472448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018513.519116:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334472448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018513.519135:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018513.519141:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018513.519149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334472448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018513.519165:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018510/real 1734018513] req@ffff8800a934d180 x1818247334472576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018513.519187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334472576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018513.519202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018513.519206:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018513.519212:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334472576:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018513.519224:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018510/real 1734018513] req@ffff8800a934ea00 x1818247334472704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018513.519243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334472704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018513.519256:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018513.519261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018513.519266:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334472704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018513.519277:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018510/real 1734018513] req@ffff8800a934c380 x1818247334472832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018513.519297:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334472832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018513.519309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018513.519313:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018513.519319:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334472832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018513.519335:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018510/real 1734018513] req@ffff8800a934d880 x1818247334472064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018513.519357:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334472064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018513.519370:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018513.519387:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018513.519393:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334472064:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018513.519405:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018510/real 1734018513] req@ffff8800a934df80 x1818247334472192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018513.519426:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334472192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018513.519439:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018513.519444:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018513.519449:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334472192:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018513.519459:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018510/real 1734018513] req@ffff8800a934c700 x1818247334472320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018513.519523:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334472320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018513.519536:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018513.519540:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018513.519545:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334472320:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018514.370740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018515.372804:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018515.520597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018515.520608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018515.520613:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018515.520624:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2782 00000100:00080000:0.0:1734018515.520633:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018515.520640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018515.520679:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334472960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018515.520720:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018515.520724:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018515.520726:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018515.520730:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2782 00000100:00080000:0.0:1734018515.520734:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018515.520738:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018515.520748:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018515.520797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334473088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018515.520807:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018515.520811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018515.520813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018515.520817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2782 00000100:00080000:0.0:1734018515.520820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018515.520823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018515.520828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018515.520839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334473216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018515.520848:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334472960:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018515.520849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018515.520853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018515.520854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018515.520858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2782 00000100:00080000:0.0:1734018515.520860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018515.520864:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018515.520869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018515.520878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334473344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018515.520886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018515.520889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018515.520890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018515.520894:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2782 00000100:00080000:0.0:1734018515.520896:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018515.520900:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018515.520904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018515.520911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334473472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734018515.520915:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018515.520919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018515.520922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018515.520923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018515.520927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2782 00000100:00080000:0.0:1734018515.520930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018515.520933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018515.520937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018515.520937:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018515/real 1734018515] req@ffff8800a934ed80 x1818247334472960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018515.520945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334473600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018515.520953:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018515.520957:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1734018515.520958:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334472960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018515.520959:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018515.520963:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2782 00000100:00080000:0.0:1734018515.520966:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018515.520968:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018515.520969:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:2.0:1734018515.520971:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000008:00100000:0.0:1734018515.520972:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018515.520976:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334472960:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018515.520980:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334473728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018515.520984:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334473088:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018515.521000:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018515.521007:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018515/real 1734018515] req@ffff8800a934ca80 x1818247334473088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018515.521019:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334473088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018515.521026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018515.521029:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018515.521032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334473088:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018515.521055:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334473216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018515.521069:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018515.521075:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018515/real 1734018515] req@ffff8800a934c700 x1818247334473216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018515.521085:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334473216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018515.521094:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018515.521097:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018515.521100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334473216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018515.521106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334473344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018515.521117:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018515.521123:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018515/real 1734018515] req@ffff8800a934df80 x1818247334473344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018515.521133:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334473344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018515.521140:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018515.521142:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018515.521146:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334473344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018515.521151:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334473472:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018515.521164:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018515.521170:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018515/real 1734018515] req@ffff8800a934d880 x1818247334473472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018515.521179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334473472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018515.521187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018515.521189:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018515.521192:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334473472:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018515.521197:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334473600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018515.521209:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018515.521215:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018515/real 1734018515] req@ffff8800a934c380 x1818247334473600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018515.521225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334473600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018515.521232:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018515.521234:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018515.521237:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334473600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018515.521242:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334473728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018515.521254:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018515.521260:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018515/real 1734018515] req@ffff8800a934ea00 x1818247334473728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018515.521270:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334473728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018515.521277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018515.521279:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018515.521282:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334473728:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018516.374708:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018517.376829:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018518.378836:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018519.380793:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018520.382766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018520.528600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018520.528611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018520.528616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018520.528627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2787 00000100:00080000:0.0:1734018520.528637:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018520.528644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018520.528709:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334473856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018520.528742:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018520.528746:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018520.528748:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018520.528753:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2787 00000100:00080000:0.0:1734018520.528756:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018520.528760:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018520.528770:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018520.528782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334473984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018520.528793:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018520.528796:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018520.528798:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018520.528802:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2787 00000100:00080000:0.0:1734018520.528804:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018520.528808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018520.528813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018520.528821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334474112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018520.528830:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018520.528833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018520.528834:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018520.528838:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2787 00000100:00080000:0.0:1734018520.528840:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018520.528843:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018520.528847:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018520.528856:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334474240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018520.528857:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334473856:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018520.528865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018520.528868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018520.528869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018520.528873:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2787 00000100:00080000:0.0:1734018520.528876:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018520.528879:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018520.528883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018520.528891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334474368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018520.528899:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018520.528902:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018520.528903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018520.528906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2787 00000100:00080000:0.0:1734018520.528909:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018520.528913:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018520.528916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018520.528924:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018520.528926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334474496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018520.528933:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018520.528936:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018520.528938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018520.528941:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2787 00000100:00080000:0.0:1734018520.528943:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018520.528947:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018520.528947:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018520/real 1734018520] req@ffff8800a934d180 x1818247334473856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018520.528951:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018520.528959:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334474624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018520.528967:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334473856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018520.528977:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018520.528980:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018520.528985:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334473856:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018520.528993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334473984:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018520.529009:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018520.529016:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018520/real 1734018520] req@ffff8800a934c000 x1818247334473984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018520.529027:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334473984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018520.529035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018520.529038:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018520.529041:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334473984:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018520.529047:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334474112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018520.529060:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018520.529066:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018520/real 1734018520] req@ffff8800a934dc00 x1818247334474112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018520.529076:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334474112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018520.529083:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018520.529086:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018520.529089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334474112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018520.529113:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334474240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018520.529126:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018520.529133:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018520/real 1734018520] req@ffff8800a934e300 x1818247334474240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018520.529143:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334474240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018520.529150:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018520.529153:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018520.529155:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334474240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018520.529161:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334474368:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018520.529177:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018520.529183:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018520/real 1734018520] req@ffff8800a934f100 x1818247334474368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018520.529193:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334474368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018520.529200:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018520.529202:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018520.529205:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334474368:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018520.529211:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334474496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018520.529226:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018520.529232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018520/real 1734018520] req@ffff8800a934fb80 x1818247334474496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018520.529242:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334474496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018520.529249:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018520.529251:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018520.529254:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334474496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018520.529259:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334474624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018520.529271:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018520.529277:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018520/real 1734018520] req@ffff8800a934d500 x1818247334474624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018520.529287:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334474624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018520.529294:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018520.529296:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018520.529299:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334474624:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018521.384752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018522.386800:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018523.388797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018524.390766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018525.392806:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018525.536613:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018525.536626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018525.536631:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018525.536642:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2792 00000100:00080000:0.0:1734018525.536652:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018525.536660:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018525.536700:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334474752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018525.536735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018525.536739:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018525.536741:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018525.536746:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2792 00000100:00080000:0.0:1734018525.536750:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018525.536753:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018525.536764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018525.536775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334474880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018525.536784:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018525.536788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018525.536789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018525.536794:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2792 00000100:00080000:0.0:1734018525.536796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018525.536800:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018525.536806:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018525.536817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334475008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018525.536826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018525.536829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018525.536831:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018525.536834:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2792 00000100:00080000:0.0:1734018525.536837:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018525.536840:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018525.536844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018525.536853:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334475136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018525.536859:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334474752:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018525.536863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018525.536865:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018525.536867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018525.536870:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2792 00000100:00080000:0.0:1734018525.536873:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018525.536876:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018525.536880:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018525.536888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334475264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018525.536895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018525.536898:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018525.536901:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018525.536905:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2792 00000100:00080000:0.0:1734018525.536907:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018525.536910:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018525.536914:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018525.536922:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334475392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018525.536930:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018525.536933:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018525.536935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018525.536938:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2792 00000100:00080000:0.0:1734018525.536941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018525.536944:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018525.536948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018525.536951:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018525.536955:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334475520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018525.536980:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018525/real 1734018525] req@ffff8800a934e680 x1818247334474752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018525.537011:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334474752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018525.537023:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018525.537027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018525.537032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334474752:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018525.537041:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334474880:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018525.537058:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018525.537066:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018525/real 1734018525] req@ffff8800a934ce00 x1818247334474880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018525.537077:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334474880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018525.537085:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018525.537088:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018525.537091:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334474880:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018525.537098:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334475008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018525.537113:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018525.537119:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018525/real 1734018525] req@ffff8800a934d500 x1818247334475008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018525.537129:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334475008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018525.537136:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018525.537140:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018525.537144:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334475008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018525.537173:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334475136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018525.537188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018525.537194:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018525/real 1734018525] req@ffff8800a934fb80 x1818247334475136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018525.537205:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334475136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018525.537213:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018525.537215:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018525.537218:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334475136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018525.537224:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334475264:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018525.537238:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018525.537244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018525/real 1734018525] req@ffff8800a934f100 x1818247334475264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018525.537254:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334475264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018525.537261:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018525.537263:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018525.537266:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334475264:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018525.537273:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334475392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018525.537289:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018525.537296:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018525/real 1734018525] req@ffff8800a934e300 x1818247334475392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018525.537307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334475392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018525.537314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018525.537316:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018525.537319:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334475392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018525.537325:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334475520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018525.537338:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018525.537344:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018525/real 1734018525] req@ffff8800a934dc00 x1818247334475520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018525.537354:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334475520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018525.537361:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018525.537364:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018525.537367:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334475520:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018526.394772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018527.396739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018528.398786:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018529.400780:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018530.402771:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018530.544591:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018530.544602:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018530.544606:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018530.544617:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2797 00000100:00080000:0.0:1734018530.544627:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018530.544634:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018530.544671:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334475648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018530.544714:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018530.544718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018530.544720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018530.544724:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2797 00000100:00080000:0.0:1734018530.544728:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018530.544731:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018530.544741:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018530.544752:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334475776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018530.544762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018530.544765:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018530.544766:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018530.544770:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2797 00000100:00080000:0.0:1734018530.544773:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018530.544777:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018530.544781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018530.544789:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334475904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018530.544798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018530.544801:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018530.544802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018530.544805:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2797 00000100:00080000:0.0:1734018530.544808:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018530.544812:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018530.544816:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018530.544820:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334475648:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018530.544823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334476032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018530.544833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018530.544836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018530.544837:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018530.544840:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2797 00000100:00080000:0.0:1734018530.544843:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018530.544846:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018530.544850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018530.544857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334476160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018530.544865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018530.544868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018530.544869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018530.544873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2797 00000100:00080000:0.0:1734018530.544875:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018530.544878:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018530.544882:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018530.544886:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018530.544892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334476288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018530.544900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018530.544904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018530.544905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018530.544909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2797 00000100:00100000:2.0:1734018530.544909:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018530/real 1734018530] req@ffff8800a934c000 x1818247334475648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018530.544912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018530.544915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018530.544919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018530.544927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334476416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018530.544936:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334475648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018530.544946:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018530.544949:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018530.544953:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334475648:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018530.544961:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334475776:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018530.544977:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018530.544984:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018530/real 1734018530] req@ffff8800a934d180 x1818247334475776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018530.544995:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334475776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018530.545002:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018530.545005:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018530.545008:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334475776:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018530.545014:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334475904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018530.545027:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018530.545033:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018530/real 1734018530] req@ffff8800a934ea00 x1818247334475904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018530.545043:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334475904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018530.545051:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018530.545053:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018530.545057:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334475904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018530.545081:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334476032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018530.545095:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018530.545101:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018530/real 1734018530] req@ffff8800a934c380 x1818247334476032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018530.545112:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334476032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018530.545119:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018530.545121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018530.545124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334476032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018530.545129:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334476160:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018530.545141:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018530.545147:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018530/real 1734018530] req@ffff8800a934d880 x1818247334476160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018530.545157:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334476160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018530.545164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018530.545166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018530.545169:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334476160:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018530.545175:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334476288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018530.545195:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018530.545201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018530/real 1734018530] req@ffff8800a934df80 x1818247334476288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018530.545211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334476288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018530.545218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018530.545220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018530.545223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334476288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018530.545228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334476416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018530.545240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018530.545246:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018530/real 1734018530] req@ffff8800a934c700 x1818247334476416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018530.545256:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334476416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018530.545263:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018530.545265:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018530.545268:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334476416:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018531.404771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018532.406799:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018533.408741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018534.410797:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018535.412767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018535.552615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018535.552626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018535.552631:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018535.552641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2802 00000100:00080000:0.0:1734018535.552650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018535.552658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018535.552695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334476544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018535.552730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018535.552734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018535.552736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018535.552741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2802 00000100:00080000:0.0:1734018535.552744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018535.552747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018535.552758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018535.552769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334476672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018535.552780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018535.552783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018535.552785:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018535.552789:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2802 00000100:00080000:0.0:1734018535.552791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018535.552795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018535.552799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018535.552810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334476800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018535.552819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018535.552822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018535.552823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018535.552827:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2802 00000100:00080000:0.0:1734018535.552829:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018535.552832:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018535.552832:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334476544:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018535.552836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018535.552845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334476928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018535.552853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018535.552856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018535.552858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018535.552861:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2802 00000100:00080000:0.0:1734018535.552864:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018535.552868:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018535.552872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018535.552880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334477056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018535.552888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018535.552890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018535.552892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018535.552896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2802 00000800:00000100:2.0:1734018535.552896:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018535.552898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018535.552902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018535.552906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018535.552913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334477184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018535.552917:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018535/real 1734018535] req@ffff8800a934ca80 x1818247334476544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018535.552921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018535.552924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018535.552925:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018535.552929:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2802 00000100:00080000:0.0:1734018535.552932:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018535.552935:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018535.552938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018535.552945:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334476544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018535.552946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334477312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018535.552955:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018535.552958:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018535.552963:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334476544:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018535.552970:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334476672:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018535.552986:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018535.552993:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018535/real 1734018535] req@ffff8800a934ed80 x1818247334476672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018535.553004:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334476672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018535.553012:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018535.553014:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018535.553018:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334476672:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018535.553041:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334476800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018535.553054:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018535.553061:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018535/real 1734018535] req@ffff8800a934c700 x1818247334476800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018535.553071:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334476800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018535.553078:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018535.553081:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018535.553084:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334476800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018535.553090:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334476928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018535.553102:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018535.553108:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018535/real 1734018535] req@ffff8800a934df80 x1818247334476928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018535.553118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334476928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018535.553125:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018535.553127:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018535.553130:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334476928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018535.553135:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334477056:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018535.553148:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018535.553154:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018535/real 1734018535] req@ffff8800a934d880 x1818247334477056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018535.553163:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334477056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018535.553171:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018535.553173:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018535.553176:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334477056:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018535.553181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334477184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018535.553196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018535.553202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018535/real 1734018535] req@ffff8800a934c380 x1818247334477184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018535.553212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334477184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018535.553219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018535.553221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018535.553224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334477184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018535.553229:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334477312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018535.553241:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018535.553247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018535/real 1734018535] req@ffff8800a934ea00 x1818247334477312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018535.553257:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334477312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018535.553264:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018535.553266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018535.553269:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334477312:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018536.414700:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018537.416744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018538.418751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018539.420730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018540.422721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018540.560555:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018540.560566:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018540.560570:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018540.560581:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2807 00000100:00080000:0.0:1734018540.560589:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018540.560597:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018540.560635:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334477440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018540.560668:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018540.560672:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018540.560674:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018540.560679:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2807 00000100:00080000:0.0:1734018540.560682:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018540.560686:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018540.560695:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018540.560706:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334477568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018540.560716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018540.560719:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018540.560721:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018540.560725:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2807 00000100:00080000:0.0:1734018540.560728:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018540.560731:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018540.560737:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018540.560745:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334477696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018540.560754:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018540.560757:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018540.560759:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018540.560762:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2807 00000100:00080000:0.0:1734018540.560764:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018540.560767:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018540.560771:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018540.560778:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334477440:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018540.560779:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334477824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018540.560788:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018540.560791:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018540.560793:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018540.560796:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2807 00000100:00080000:0.0:1734018540.560798:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018540.560802:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018540.560806:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018540.560813:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334477952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018540.560821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018540.560824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018540.560826:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018540.560830:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2807 00000100:00080000:0.0:1734018540.560833:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018540.560836:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018540.560841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018540.560846:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018540.560850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334478080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018540.560858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018540.560861:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018540.560862:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018540.560865:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2807 00000100:00080000:0.0:1734018540.560868:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018540.560870:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018540/real 1734018540] req@ffff8800a934d180 x1818247334477440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018540.560871:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018540.560875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018540.560882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334478208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018540.560898:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334477440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018540.560908:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018540.560911:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018540.560916:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334477440:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018540.560923:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334477568:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018540.560939:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018540.560946:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018540/real 1734018540] req@ffff8800a934c000 x1818247334477568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018540.560957:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334477568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018540.560964:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018540.560967:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018540.560970:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334477568:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018540.560976:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334477696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018540.560989:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018540.560995:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018540/real 1734018540] req@ffff8800a934dc00 x1818247334477696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018540.561005:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334477696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018540.561013:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018540.561015:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018540.561019:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334477696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018540.561042:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334477824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018540.561056:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018540.561062:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018540/real 1734018540] req@ffff8800a934e300 x1818247334477824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018540.561072:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334477824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018540.561080:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018540.561082:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018540.561085:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334477824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018540.561090:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334477952:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018540.561102:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018540.561108:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018540/real 1734018540] req@ffff8800a934f100 x1818247334477952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018540.561118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334477952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018540.561125:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018540.561127:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018540.561131:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334477952:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018540.561136:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334478080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018540.561151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018540.561157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018540/real 1734018540] req@ffff8800a934fb80 x1818247334478080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018540.561167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334478080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018540.561174:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018540.561177:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018540.561179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334478080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018540.561184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334478208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018540.561197:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018540.561202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018540/real 1734018540] req@ffff8800a934d500 x1818247334478208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018540.561212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334478208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018540.561219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018540.561221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018540.561224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334478208:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018541.424731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018542.426661:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018543.428798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018544.430761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018545.432789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018545.568573:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018545.568585:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018545.568589:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018545.568601:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2812 00000100:00080000:0.0:1734018545.568610:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018545.568618:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018545.568656:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334478336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018545.568698:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018545.568702:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018545.568704:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018545.568708:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2812 00000100:00080000:0.0:1734018545.568712:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018545.568716:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018545.568725:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018545.568737:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334478464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018545.568746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018545.568749:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018545.568751:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018545.568755:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2812 00000100:00080000:0.0:1734018545.568757:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018545.568761:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018545.568766:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018545.568776:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334478592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018545.568784:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018545.568787:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018545.568789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018545.568792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2812 00000100:00080000:0.0:1734018545.568794:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018545.568798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018545.568802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018545.568811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334478720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018545.568812:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334478336:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018545.568820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018545.568823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018545.568824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018545.568827:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2812 00000100:00080000:0.0:1734018545.568830:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018545.568833:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018545.568837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018545.568844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334478848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018545.568852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018545.568855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018545.568856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018545.568859:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2812 00000100:00080000:0.0:1734018545.568862:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018545.568865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018545.568869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018545.568877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334478976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734018545.568877:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018545.568885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018545.568889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018545.568891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018545.568894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2812 00000100:00080000:0.0:1734018545.568897:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018545.568899:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018545/real 1734018545] req@ffff8800a934ce00 x1818247334478336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018545.568900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018545.568904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018545.568911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334479104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018545.568928:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334478336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018545.568938:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018545.568941:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018545.568946:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334478336:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018545.568953:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334478464:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018545.568970:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018545.568977:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018545/real 1734018545] req@ffff8800a934e680 x1818247334478464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018545.568988:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334478464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018545.568996:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018545.568998:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018545.569001:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334478464:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018545.569007:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334478592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018545.569020:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018545.569026:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018545/real 1734018545] req@ffff8800a934d500 x1818247334478592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018545.569036:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334478592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018545.569044:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018545.569047:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018545.569050:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334478592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018545.569074:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334478720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018545.569087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018545.569093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018545/real 1734018545] req@ffff8800a934fb80 x1818247334478720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018545.569104:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334478720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018545.569111:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018545.569113:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018545.569116:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334478720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018545.569122:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334478848:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018545.569134:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018545.569140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018545/real 1734018545] req@ffff8800a934f100 x1818247334478848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018545.569150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334478848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018545.569157:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018545.569159:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018545.569162:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334478848:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018545.569168:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334478976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018545.569183:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018545.569190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018545/real 1734018545] req@ffff8800a934e300 x1818247334478976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018545.569200:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334478976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018545.569207:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018545.569210:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018545.569213:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334478976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018545.569218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334479104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018545.569230:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018545.569235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018545/real 1734018545] req@ffff8800a934dc00 x1818247334479104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018545.569280:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334479104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018545.569288:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018545.569290:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018545.569293:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334479104:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018546.434731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018547.436771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018548.438721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018549.440791:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018550.442749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018550.576571:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018550.576583:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018550.576587:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018550.576598:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2817 00000100:00080000:0.0:1734018550.576608:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018550.576615:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018550.576653:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334479232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018550.576687:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018550.576691:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018550.576693:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018550.576698:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2817 00000100:00080000:0.0:1734018550.576701:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018550.576705:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018550.576715:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018550.576726:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334479360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018550.576737:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018550.576741:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018550.576742:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018550.576746:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2817 00000100:00080000:0.0:1734018550.576749:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018550.576752:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018550.576757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018550.576765:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334479488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018550.576774:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018550.576777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018550.576779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018550.576782:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2817 00000100:00080000:0.0:1734018550.576784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018550.576787:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018550.576791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018550.576793:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334479232:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018550.576799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334479616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018550.576808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018550.576811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018550.576812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018550.576816:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2817 00000100:00080000:0.0:1734018550.576819:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018550.576822:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018550.576826:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018550.576834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334479744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018550.576842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018550.576845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018550.576846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018550.576849:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2817 00000100:00080000:0.0:1734018550.576852:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018550.576856:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018550.576860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018550.576860:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018550.576869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334479872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018550.576877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018550.576880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018550.576881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018550.576882:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018550/real 1734018550] req@ffff8800a934c000 x1818247334479232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018550.576885:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2817 00000100:00080000:0.0:1734018550.576887:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018550.576890:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018550.576894:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018550.576902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334480000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018550.576909:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334479232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018550.576920:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018550.576923:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018550.576927:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334479232:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018550.576935:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334479360:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018550.576951:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018550.576958:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018550/real 1734018550] req@ffff8800a934d180 x1818247334479360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018550.576969:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334479360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018550.576977:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018550.576979:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018550.576983:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334479360:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018550.576988:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334479488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018550.577001:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018550.577007:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018550/real 1734018550] req@ffff8800a934ea00 x1818247334479488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018550.577017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334479488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018550.577025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018550.577028:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018550.577031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334479488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018550.577055:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334479616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018550.577069:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018550.577075:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018550/real 1734018550] req@ffff8800a934c380 x1818247334479616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018550.577085:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334479616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018550.577093:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018550.577095:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018550.577098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334479616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018550.577103:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334479744:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018550.577115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018550.577121:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018550/real 1734018550] req@ffff8800a934d880 x1818247334479744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018550.577131:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334479744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018550.577138:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018550.577140:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018550.577143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334479744:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018550.577149:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334479872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018550.577215:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018550.577224:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018550/real 1734018550] req@ffff8800a934df80 x1818247334479872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018550.577235:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334479872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018550.577242:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018550.577245:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018550.577248:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334479872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018550.577253:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334480000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018550.577266:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018550.577272:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018550/real 1734018550] req@ffff8800a934c700 x1818247334480000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018550.577282:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334480000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018550.577289:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018550.577292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018550.577294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334480000:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018551.444749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018552.446737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018553.448791:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018554.450742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018555.452789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018555.584597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018555.584608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018555.584612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018555.584624:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2822 00000100:00080000:0.0:1734018555.584632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018555.584640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018555.584677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334480128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018555.584711:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018555.584715:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018555.584717:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018555.584722:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2822 00000100:00080000:0.0:1734018555.584725:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018555.584729:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018555.584739:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018555.584750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334480256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018555.584760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018555.584763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018555.584765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018555.584769:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2822 00000100:00080000:0.0:1734018555.584772:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018555.584775:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018555.584781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018555.584792:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334480384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018555.584801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018555.584804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018555.584806:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018555.584809:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2822 00000100:00080000:0.0:1734018555.584812:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018555.584815:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018555.584819:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018555.584819:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334480128:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018555.584827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334480512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018555.584836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018555.584839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018555.584841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018555.584844:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2822 00000100:00080000:0.0:1734018555.584846:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018555.584849:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018555.584853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018555.584862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334480640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018555.584870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018555.584872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018555.584875:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018555.584878:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2822 00000100:00080000:0.0:1734018555.584881:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734018555.584884:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018555.584885:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018555.584889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018555.584896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334480768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018555.584904:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734018555.584906:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018555/real 1734018555] req@ffff8800a934ed80 x1818247334480128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018555.584907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018555.584909:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018555.584912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2822 00000100:00080000:0.0:1734018555.584915:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018555.584918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018555.584922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018555.584929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334480896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018555.584934:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334480128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018555.584944:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018555.584947:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018555.584951:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334480128:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018555.584959:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334480256:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018555.584974:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018555.584982:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018555/real 1734018555] req@ffff8800a934ca80 x1818247334480256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018555.584993:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334480256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018555.585000:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018555.585003:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018555.585006:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334480256:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018555.585012:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334480384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018555.585025:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018555.585031:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018555/real 1734018555] req@ffff8800a934c700 x1818247334480384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018555.585041:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334480384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018555.585048:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018555.585051:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018555.585055:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334480384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018555.585121:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334480512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018555.585139:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018555.585146:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018555/real 1734018555] req@ffff8800a934df80 x1818247334480512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018555.585157:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334480512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018555.585165:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018555.585167:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018555.585170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334480512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018555.585175:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334480640:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018555.585188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018555.585193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018555/real 1734018555] req@ffff8800a934d880 x1818247334480640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018555.585204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334480640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018555.585212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018555.585214:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018555.585217:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334480640:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018555.585222:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334480768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018555.585237:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018555.585243:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018555/real 1734018555] req@ffff8800a934c380 x1818247334480768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018555.585253:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334480768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018555.585260:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018555.585263:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018555.585265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334480768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018555.585270:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334480896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018555.585282:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018555.585288:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018555/real 1734018555] req@ffff8800a934ea00 x1818247334480896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018555.585298:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334480896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018555.585305:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018555.585307:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018555.585310:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334480896:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018556.454743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018557.456795:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018558.458781:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018559.460811:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018560.462758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018560.592625:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018560.592637:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018560.592641:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018560.592653:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2827 00000100:00080000:0.0:1734018560.592662:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018560.592669:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018560.592706:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334481024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018560.592748:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018560.592752:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018560.592754:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018560.592759:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2827 00000100:00080000:0.0:1734018560.592762:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018560.592766:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018560.592775:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018560.592787:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334481152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018560.592796:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018560.592799:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018560.592800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018560.592804:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2827 00000100:00080000:0.0:1734018560.592807:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018560.592811:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018560.592815:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018560.592823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334481280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018560.592832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018560.592835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018560.592836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018560.592839:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2827 00000100:00080000:0.0:1734018560.592842:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018560.592846:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018560.592850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018560.592854:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334481024:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018560.592858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334481408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018560.592867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018560.592870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018560.592871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018560.592874:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2827 00000100:00080000:0.0:1734018560.592877:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018560.592880:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018560.592884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018560.592891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334481536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018560.592899:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018560.592901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018560.592903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018560.592906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2827 00000100:00080000:0.0:1734018560.592909:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018560.592912:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018560.592916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018560.592919:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018560.592925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334481664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018560.592933:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018560.592937:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018560.592938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018560.592942:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2827 00000100:00100000:2.0:1734018560.592942:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018560/real 1734018560] req@ffff8800a934d180 x1818247334481024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018560.592945:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018560.592948:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018560.592952:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018560.592959:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334481792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018560.592969:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334481024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018560.592979:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018560.592982:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018560.593027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334481024:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018560.593035:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334481152:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018560.593055:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018560.593063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018560/real 1734018560] req@ffff8800a934c000 x1818247334481152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018560.593074:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334481152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018560.593082:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018560.593084:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018560.593088:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334481152:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018560.593094:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334481280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018560.593107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018560.593113:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018560/real 1734018560] req@ffff8800a934dc00 x1818247334481280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018560.593123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334481280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018560.593131:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018560.593134:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018560.593137:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334481280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018560.593160:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334481408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018560.593174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018560.593180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018560/real 1734018560] req@ffff8800a934e300 x1818247334481408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018560.593190:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334481408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018560.593197:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018560.593199:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018560.593202:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334481408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018560.593208:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334481536:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018560.593219:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018560.593225:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018560/real 1734018560] req@ffff8800a934f100 x1818247334481536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018560.593235:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334481536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018560.593243:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018560.593245:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018560.593248:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334481536:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018560.593253:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334481664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018560.593273:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018560.593279:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018560/real 1734018560] req@ffff8800a934fb80 x1818247334481664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018560.593289:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334481664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018560.593296:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018560.593298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018560.593301:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334481664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018560.593307:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334481792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018560.593318:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018560.593324:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018560/real 1734018560] req@ffff8800a934d500 x1818247334481792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018560.593334:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334481792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018560.593341:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018560.593343:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018560.593346:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334481792:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018561.464731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018562.466734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018563.468783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018564.470749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018565.472764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018565.600588:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018565.600598:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018565.600603:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018565.600614:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2832 00000100:00080000:0.0:1734018565.600623:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018565.600631:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018565.600668:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334481920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018565.600701:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018565.600705:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018565.600707:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018565.600712:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2832 00000100:00080000:0.0:1734018565.600715:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018565.600719:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018565.600729:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018565.600740:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334482048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018565.600752:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018565.600755:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018565.600757:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018565.600761:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2832 00000100:00080000:0.0:1734018565.600763:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018565.600767:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018565.600771:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018565.600781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334482176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018565.600791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018565.600794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018565.600795:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018565.600799:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2832 00000100:00080000:0.0:1734018565.600801:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018565.600804:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018565.600805:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334481920:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018565.600809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018565.600817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334482304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018565.600825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018565.600828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018565.600829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018565.600833:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2832 00000100:00080000:0.0:1734018565.600836:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018565.600839:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018565.600843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018565.600851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334482432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018565.600859:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018565.600861:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018565.600863:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018565.600866:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2832 00000100:00080000:0.0:1734018565.600869:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734018565.600871:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018565.600872:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018565.600876:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018565.600884:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334482560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018565.600892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018565.600894:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1734018565.600894:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018565/real 1734018565] req@ffff8800a934e680 x1818247334481920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018565.600896:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018565.600939:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2832 00000100:00080000:0.0:1734018565.600943:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018565.600946:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018565.600950:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018565.600954:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334481920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018565.600959:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334482688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018565.600964:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018565.600967:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018565.600973:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334481920:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018565.600980:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334482048:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018565.600999:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018565.601007:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018565/real 1734018565] req@ffff8800a934ce00 x1818247334482048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018565.601018:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334482048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018565.601026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018565.601029:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018565.601032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334482048:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018565.601056:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334482176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018565.601069:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018565.601076:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018565/real 1734018565] req@ffff8800a934d500 x1818247334482176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018565.601086:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334482176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018565.601094:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018565.601097:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018565.601100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334482176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018565.601106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334482304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018565.601118:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018565.601124:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018565/real 1734018565] req@ffff8800a934fb80 x1818247334482304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018565.601134:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334482304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018565.601142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018565.601144:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018565.601147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334482304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018565.601153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334482432:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018565.601165:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018565.601171:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018565/real 1734018565] req@ffff8800a934f100 x1818247334482432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018565.601181:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334482432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018565.601189:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018565.601191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018565.601194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334482432:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018565.601199:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334482560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018565.601220:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018565.601226:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018565/real 1734018565] req@ffff8800a934e300 x1818247334482560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018565.601237:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334482560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018565.601244:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018565.601246:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018565.601249:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334482560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018565.601255:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334482688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018565.601267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018565.601273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018565/real 1734018565] req@ffff8800a934dc00 x1818247334482688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018565.601283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334482688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018565.601290:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018565.601292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018565.601295:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334482688:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018566.474788:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018567.476768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018568.478749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018569.480732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018570.482780:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018570.608598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018570.608610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018570.608614:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018570.608625:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2837 00000100:00080000:0.0:1734018570.608634:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018570.608642:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018570.608679:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334482816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018570.608712:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018570.608716:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018570.608718:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018570.608722:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2837 00000100:00080000:0.0:1734018570.608725:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018570.608729:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018570.608739:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018570.608750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334482944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018570.608760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018570.608763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018570.608765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018570.608769:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2837 00000100:00080000:0.0:1734018570.608771:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018570.608775:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018570.608781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018570.608790:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334483072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018570.608798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018570.608802:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018570.608803:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018570.608806:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2837 00000100:00080000:0.0:1734018570.608809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018570.608812:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018570.608847:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334482816:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018570.608852:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018570.608862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334483200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018570.608871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018570.608874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018570.608876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018570.608879:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2837 00000100:00080000:0.0:1734018570.608882:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018570.608885:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018570.608889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018570.608899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334483328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018570.608907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018570.608910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018570.608913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1734018570.608914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018570.608916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2837 00000100:00080000:0.0:1734018570.608919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018570.608922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018570.608926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018570.608934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334483456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018570.608936:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018570/real 1734018570] req@ffff8800a934c000 x1818247334482816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018570.608942:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018570.608945:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018570.608947:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018570.608950:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2837 00000100:00080000:0.0:1734018570.608952:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018570.608956:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018570.608959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018570.608964:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334482816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018570.608967:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334483584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018570.608974:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018570.608978:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018570.608982:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334482816:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018570.608990:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334482944:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018570.609007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018570.609014:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018570/real 1734018570] req@ffff8800a934d180 x1818247334482944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018570.609025:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334482944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018570.609033:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018570.609035:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018570.609038:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334482944:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018570.609044:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334483072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018570.609057:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018570.609063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018570/real 1734018570] req@ffff8800a934ea00 x1818247334483072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018570.609073:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334483072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018570.609081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018570.609083:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018570.609086:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334483072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018570.609111:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334483200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018570.609124:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018570.609130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018570/real 1734018570] req@ffff8800a934c380 x1818247334483200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018570.609140:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334483200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018570.609148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018570.609150:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018570.609153:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334483200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018570.609158:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334483328:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018570.609170:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018570.609176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018570/real 1734018570] req@ffff8800a934d880 x1818247334483328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018570.609186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334483328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018570.609193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018570.609195:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018570.609198:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334483328:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018570.609204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334483456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018570.609224:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018570.609230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018570/real 1734018570] req@ffff8800a934df80 x1818247334483456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018570.609240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334483456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018570.609247:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018570.609249:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018570.609252:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334483456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018570.609257:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334483584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018570.609269:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018570.609275:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018570/real 1734018570] req@ffff8800a934c700 x1818247334483584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018570.609285:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334483584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018570.609292:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018570.609294:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018570.609297:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334483584:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018571.484759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018572.486741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018573.488809:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018574.490822:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018575.492793:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018575.616580:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018575.616592:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018575.616596:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018575.616607:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2842 00000100:00080000:0.0:1734018575.616616:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018575.616624:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018575.616661:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334483712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018575.616703:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018575.616707:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018575.616709:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018575.616713:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2842 00000100:00080000:0.0:1734018575.616717:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018575.616720:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018575.616763:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018575.616777:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334483840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018575.616801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018575.616803:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018575.616804:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018575.616806:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2842 00000100:00080000:0.0:1734018575.616808:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018575.616811:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018575.616814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018575.616821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334483968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018575.616826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018575.616828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018575.616829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018575.616831:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2842 00000100:00080000:0.0:1734018575.616833:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018575.616835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018575.616838:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018575.616838:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334483712:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018575.616843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334484096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018575.616848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018575.616850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018575.616851:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018575.616853:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2842 00000100:00080000:0.0:1734018575.616854:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018575.616856:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018575.616858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018575.616863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334484224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018575.616868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018575.616870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018575.616871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018575.616873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2842 00000100:00080000:0.0:1734018575.616874:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018575.616876:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018575.616879:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018575.616883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334484352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018575.616888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018575.616890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018575.616891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018575.616893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2842 00000100:00080000:0.0:1734018575.616895:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018575.616897:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018575.616899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018575.616902:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334483840:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018575.616904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334484480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018575.616915:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334483968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018575.616930:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334484096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018575.616940:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334484224:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018575.616949:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334484352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018575.616958:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334484480:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018576.494787:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018577.496806:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018578.498862:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018578.622828:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734018578.622887:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018578.622914:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018578.622957:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018578.622967:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018578.622975:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018578.622984:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018578.622997:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018578.623009:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:2.0:1734018578.623070:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018575/real 1734018578] req@ffff8800a934df80 x1818247334484096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018578.623105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334484096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018578.623117:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018578.623121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018578.623126:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334484096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018578.623137:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018575/real 1734018578] req@ffff8800a934d880 x1818247334484224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018578.623156:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334484224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018578.623169:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018578.623173:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018578.623178:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334484224:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018578.623189:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018575/real 1734018578] req@ffff8800a934c380 x1818247334484352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018578.623207:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334484352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018578.623220:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018578.623225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018578.623230:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334484352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018578.623240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018575/real 1734018578] req@ffff8800a934ea00 x1818247334484480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018578.623259:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334484480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018578.623271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018578.623275:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018578.623280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334484480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018578.623292:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018575/real 1734018578] req@ffff8800a934ca80 x1818247334483712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018578.623311:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334483712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018578.623323:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018578.623327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018578.623331:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334483712:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018578.623342:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018575/real 1734018578] req@ffff8800a934ed80 x1818247334483840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018578.623361:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334483840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018578.623383:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018578.623387:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018578.623392:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334483840:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018578.623403:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018575/real 1734018578] req@ffff8800a934c700 x1818247334483968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018578.623421:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334483968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018578.623434:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018578.623438:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018578.623442:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334483968:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018579.500800:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018580.502746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018580.624609:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018580.624620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018580.624625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018580.624637:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2847 00000100:00080000:0.0:1734018580.624671:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018580.624680:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018580.624717:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334484608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018580.624750:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018580.624754:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018580.624756:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018580.624761:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2847 00000100:00080000:0.0:1734018580.624764:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018580.624768:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018580.624778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018580.624789:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334484736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018580.624800:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018580.624804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018580.624806:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018580.624809:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2847 00000100:00080000:0.0:1734018580.624812:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018580.624816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018580.624821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018580.624829:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334484864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018580.624838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018580.624841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018580.624842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018580.624845:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2847 00000100:00080000:0.0:1734018580.624848:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018580.624851:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018580.624855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018580.624855:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334484608:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018580.624863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334484992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018580.624872:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018580.624875:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018580.624877:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018580.624880:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2847 00000100:00080000:0.0:1734018580.624883:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018580.624887:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018580.624891:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018580.624900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334485120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018580.624908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018580.624910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018580.624912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018580.624915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2847 00000100:00080000:0.0:1734018580.624918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018580.624921:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734018580.624922:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018580.624925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018580.624933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334485248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018580.624940:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018580.624944:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018580.624945:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018580.624945:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018580/real 1734018580] req@ffff8800a934d180 x1818247334484608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018580.624949:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2847 00000100:00080000:0.0:1734018580.624951:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018580.624954:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018580.624958:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018580.624965:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334485376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018580.624965:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334484608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018580.624975:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018580.624978:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018580.624983:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334484608:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018580.624991:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334484736:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018580.625007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018580.625014:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018580/real 1734018580] req@ffff8800a934c000 x1818247334484736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018580.625026:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334484736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018580.625033:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018580.625036:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018580.625039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334484736:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018580.625045:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334484864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018580.625058:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018580.625064:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018580/real 1734018580] req@ffff8800a934dc00 x1818247334484864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018580.625074:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334484864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018580.625082:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018580.625084:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018580.625087:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334484864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018580.625113:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334484992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018580.625126:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018580.625133:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018580/real 1734018580] req@ffff8800a934e300 x1818247334484992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018580.625144:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334484992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018580.625158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018580.625161:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018580.625164:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334484992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018580.625169:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334485120:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018580.625186:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018580.625192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018580/real 1734018580] req@ffff8800a934f100 x1818247334485120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018580.625202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334485120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018580.625209:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018580.625211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018580.625214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334485120:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018580.625219:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334485248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018580.625231:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018580.625237:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018580/real 1734018580] req@ffff8800a934fb80 x1818247334485248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018580.625247:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334485248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018580.625254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018580.625257:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018580.625260:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334485248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018580.625265:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334485376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018580.625278:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018580.625284:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018580/real 1734018580] req@ffff8800a934d500 x1818247334485376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018580.625294:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334485376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018580.625301:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018580.625303:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018580.625306:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334485376:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018581.504751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018582.506736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018583.508807:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018584.510752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018585.512801:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018585.632617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018585.632631:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018585.632636:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018585.632647:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2852 00000100:00080000:0.0:1734018585.632656:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018585.632664:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018585.632702:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334485504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018585.632734:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018585.632738:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018585.632740:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018585.632744:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2852 00000100:00080000:0.0:1734018585.632747:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018585.632751:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018585.632762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018585.632773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334485632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018585.632783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018585.632786:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018585.632788:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018585.632792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2852 00000100:00080000:0.0:1734018585.632795:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018585.632798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018585.632805:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018585.632815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334485760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018585.632824:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018585.632827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018585.632829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018585.632832:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2852 00000100:00080000:0.0:1734018585.632835:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018585.632838:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018585.632840:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334485504:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018585.632842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018585.632850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334485888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018585.632859:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018585.632862:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018585.632863:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018585.632867:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2852 00000100:00080000:0.0:1734018585.632870:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018585.632873:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018585.632877:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018585.632884:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334486016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018585.632892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018585.632895:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018585.632897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018585.632900:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2852 00000100:00080000:0.0:1734018585.632903:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734018585.632905:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018585.632907:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018585.632911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018585.632919:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334486144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018585.632926:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734018585.632928:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018585/real 1734018585] req@ffff8800a934ce00 x1818247334485504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018585.632929:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018585.632931:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018585.632934:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2852 00000100:00080000:0.0:1734018585.632937:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018585.632940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018585.632944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018585.632949:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334485504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018585.632952:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334486272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018585.632960:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018585.632963:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018585.632968:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334485504:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018585.632975:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334485632:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018585.632991:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018585.632998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018585/real 1734018585] req@ffff8800a934e680 x1818247334485632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018585.633017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334485632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018585.633025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018585.633028:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018585.633031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334485632:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018585.633037:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334485760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018585.633050:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018585.633056:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018585/real 1734018585] req@ffff8800a934d500 x1818247334485760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018585.633066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334485760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018585.633074:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018585.633076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018585.633079:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334485760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018585.633102:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334485888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018585.633115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018585.633121:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018585/real 1734018585] req@ffff8800a934fb80 x1818247334485888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018585.633132:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334485888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018585.633139:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018585.633142:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018585.633145:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334485888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018585.633151:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334486016:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018585.633163:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018585.633169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018585/real 1734018585] req@ffff8800a934f100 x1818247334486016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018585.633179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334486016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018585.633186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018585.633189:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018585.633192:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334486016:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018585.633197:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334486144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018585.633209:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018585.633215:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018585/real 1734018585] req@ffff8800a934e300 x1818247334486144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018585.633225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334486144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018585.633232:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018585.633234:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018585.633237:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334486144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018585.633242:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334486272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018585.633257:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018585.633263:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018585/real 1734018585] req@ffff8800a934dc00 x1818247334486272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018585.633272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334486272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018585.633280:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018585.633282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018585.633285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334486272:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018586.514732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018587.516805:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018588.518740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018589.520775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018590.522744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018590.640598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018590.640608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018590.640614:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018590.640627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2857 00000100:00080000:0.0:1734018590.640638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018590.640648:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018590.640695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334486400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018590.640739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018590.640744:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018590.640746:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018590.640750:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2857 00000100:00080000:0.0:1734018590.640754:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018590.640758:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018590.640769:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018590.640781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334486528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018590.640790:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018590.640794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018590.640795:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018590.640800:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2857 00000100:00080000:0.0:1734018590.640803:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018590.640807:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018590.640812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018590.640821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334486656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018590.640829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018590.640833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018590.640834:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018590.640838:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2857 00000100:00080000:0.0:1734018590.640840:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018590.640845:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018590.640850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018590.640851:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334486400:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018590.640858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334486784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018590.640867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018590.640871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018590.640872:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018590.640876:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2857 00000100:00080000:0.0:1734018590.640878:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018590.640881:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018590.640885:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018590.640893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334486912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018590.640901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018590.640904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018590.640906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018590.640909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2857 00000100:00080000:0.0:1734018590.640912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018590.640916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018590.640920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018590.640930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334487040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734018590.640938:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018590.640939:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018590.640943:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018590.640944:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018590.640948:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2857 00000100:00080000:0.0:1734018590.640951:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018590.640955:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018590.640959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018590.640966:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018590/real 1734018590] req@ffff8800a934c000 x1818247334486400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018590.640967:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334487168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018590.640988:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334486400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018590.641000:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018590.641004:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018590.641009:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334486400:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018590.641017:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334486528:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018590.641035:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018590.641042:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018590/real 1734018590] req@ffff8800a934d180 x1818247334486528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018590.641053:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334486528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018590.641062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018590.641065:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018590.641068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334486528:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018590.641074:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334486656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018590.641088:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018590.641094:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018590/real 1734018590] req@ffff8800a934c700 x1818247334486656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018590.641104:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334486656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018590.641112:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018590.641114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018590.641118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334486656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018590.641147:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334486784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018590.641161:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018590.641167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018590/real 1734018590] req@ffff8800a934ed80 x1818247334486784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018590.641178:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334486784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018590.641186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018590.641188:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018590.641191:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334486784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018590.641197:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334486912:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018590.641210:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018590.641216:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018590/real 1734018590] req@ffff8800a934ca80 x1818247334486912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018590.641226:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334486912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018590.641234:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018590.641236:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018590.641239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334486912:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018590.641244:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334487040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018590.641256:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018590.641262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018590/real 1734018590] req@ffff8800a934ea00 x1818247334487040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018590.641272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334487040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018590.641279:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018590.641281:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018590.641284:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334487040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018590.641289:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334487168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018590.641302:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018590.641308:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018590/real 1734018590] req@ffff8800a934c380 x1818247334487168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018590.641319:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334487168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018590.641327:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018590.641329:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018590.641332:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334487168:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018591.524753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018592.526760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018593.528803:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018594.530761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018595.532769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018595.648613:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018595.648624:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018595.648629:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018595.648640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2862 00000100:00080000:0.0:1734018595.648649:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018595.648657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018595.648694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334487296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018595.648727:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018595.648731:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018595.648733:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018595.648737:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2862 00000100:00080000:0.0:1734018595.648740:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018595.648744:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018595.648755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018595.648766:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334487424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018595.648777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018595.648780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018595.648781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018595.648785:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2862 00000100:00080000:0.0:1734018595.648787:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018595.648791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018595.648795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018595.648806:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334487552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018595.648815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018595.648818:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018595.648820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018595.648823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2862 00000100:00080000:0.0:1734018595.648825:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018595.648828:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018595.648833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018595.648837:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334487296:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018595.648841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334487680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018595.648850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018595.648853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018595.648855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018595.648858:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2862 00000100:00080000:0.0:1734018595.648861:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018595.648864:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018595.648869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018595.648876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334487808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018595.648884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018595.648887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018595.648888:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018595.648892:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2862 00000100:00080000:0.0:1734018595.648895:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018595.648898:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018595.648902:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018595.648903:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018595.648910:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334487936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018595.648918:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018595.648921:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018595.648923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018595.648925:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018595/real 1734018595] req@ffff8800a934d880 x1818247334487296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018595.648926:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2862 00000100:00080000:0.0:1734018595.648929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018595.648932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018595.648935:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018595.648943:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334488064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018595.648946:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334487296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018595.648957:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018595.648960:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018595.648965:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334487296:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018595.648973:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334487424:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018595.648989:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018595.648996:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018595/real 1734018595] req@ffff8800a934df80 x1818247334487424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018595.649007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334487424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018595.649015:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018595.649018:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018595.649022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334487424:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018595.649028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334487552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018595.649042:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018595.649048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018595/real 1734018595] req@ffff8800a934c380 x1818247334487552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018595.649058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334487552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018595.649066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018595.649068:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018595.649072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334487552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018595.649095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334487680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018595.649108:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018595.649114:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018595/real 1734018595] req@ffff8800a934ea00 x1818247334487680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018595.649125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334487680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018595.649132:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018595.649135:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018595.649138:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334487680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018595.649143:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334487808:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018595.649159:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018595.649165:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018595/real 1734018595] req@ffff8800a934ca80 x1818247334487808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018595.649176:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334487808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018595.649183:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018595.649185:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018595.649188:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334487808:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018595.649193:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334487936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018595.649205:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018595.649211:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018595/real 1734018595] req@ffff8800a934ed80 x1818247334487936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018595.649221:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334487936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018595.649228:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018595.649230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018595.649233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334487936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018595.649238:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334488064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018595.649251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018595.649257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018595/real 1734018595] req@ffff8800a934c700 x1818247334488064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018595.649273:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334488064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018595.649280:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018595.649282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018595.649285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334488064:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018596.534713:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018597.536755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018598.538739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018599.540815:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018600.542745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018600.656619:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018600.656630:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018600.656635:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018600.656646:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2867 00000100:00080000:0.0:1734018600.656655:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018600.656663:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018600.656701:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334488192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018600.656735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018600.656739:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018600.656741:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018600.656746:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2867 00000100:00080000:0.0:1734018600.656749:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018600.656753:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018600.656763:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018600.656774:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334488320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018600.656784:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018600.656787:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018600.656788:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018600.656793:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2867 00000100:00080000:0.0:1734018600.656795:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018600.656799:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018600.656805:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018600.656813:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334488448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018600.656822:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018600.656825:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018600.656827:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018600.656830:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2867 00000100:00080000:0.0:1734018600.656833:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018600.656836:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018600.656840:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018600.656846:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334488192:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018600.656847:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334488576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018600.656856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018600.656859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018600.656861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018600.656864:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2867 00000100:00080000:0.0:1734018600.656866:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018600.656870:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018600.656874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018600.656881:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334488704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018600.656889:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018600.656892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018600.656894:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018600.656898:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2867 00000100:00080000:0.0:1734018600.656901:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018600.656904:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018600.656908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018600.656909:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018600.656918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334488832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018600.656925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018600.656928:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018600.656930:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018600.656931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018600/real 1734018600] req@ffff8800a934d180 x1818247334488192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018600.656933:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2867 00000100:00080000:0.0:1734018600.656936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018600.656939:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018600.656943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018600.656950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334488960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018600.656950:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334488192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018600.656961:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018600.656964:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018600.656969:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334488192:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018600.656977:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334488320:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018600.656993:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018600.657000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018600/real 1734018600] req@ffff8800a934c000 x1818247334488320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018600.657011:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334488320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018600.657018:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018600.657022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018600.657025:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334488320:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018600.657032:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334488448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018600.657045:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018600.657051:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018600/real 1734018600] req@ffff8800a934dc00 x1818247334488448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018600.657061:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334488448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018600.657069:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018600.657072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018600.657075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334488448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018600.657099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334488576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018600.657112:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018600.657118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018600/real 1734018600] req@ffff8800a934e300 x1818247334488576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018600.657129:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334488576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018600.657136:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018600.657138:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018600.657141:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334488576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018600.657146:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334488704:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018600.657167:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018600.657173:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018600/real 1734018600] req@ffff8800a934f100 x1818247334488704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018600.657184:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334488704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018600.657191:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018600.657194:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018600.657196:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334488704:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018600.657202:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334488832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018600.657213:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018600.657219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018600/real 1734018600] req@ffff8800a934fb80 x1818247334488832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018600.657229:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334488832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018600.657236:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018600.657238:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018600.657241:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334488832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018600.657246:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334488960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018600.657258:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018600.657268:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018600/real 1734018600] req@ffff8800a934d500 x1818247334488960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018600.657284:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334488960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018600.657291:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018600.657293:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018600.657296:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334488960:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018601.544727:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018602.546649:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018603.548764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018604.550751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018605.552760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018605.664542:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018605.664554:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018605.664558:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018605.664569:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2872 00000100:00080000:0.0:1734018605.664578:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018605.664586:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018605.664622:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334489088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018605.664663:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018605.664667:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018605.664669:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018605.664674:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2872 00000100:00080000:0.0:1734018605.664677:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018605.664681:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018605.664691:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018605.664702:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334489216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018605.664711:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018605.664714:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018605.664716:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018605.664720:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2872 00000100:00080000:0.0:1734018605.664723:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018605.664727:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018605.664732:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018605.664742:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334489344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018605.664751:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018605.664754:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018605.664755:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018605.664759:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2872 00000100:00080000:0.0:1734018605.664761:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018605.664765:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018605.664769:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334489088:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018605.664770:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018605.664778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334489472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018605.664787:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018605.664790:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018605.664791:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018605.664795:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2872 00000100:00080000:0.0:1734018605.664797:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018605.664800:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018605.664804:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018605.664812:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334489600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018605.664820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018605.664822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018605.664824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018605.664827:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2872 00000100:00080000:0.0:1734018605.664830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018605.664833:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734018605.664835:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734018605.664837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018605.664845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334489728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018605.664853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018605.664856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018605.664858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018605.664859:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018605/real 1734018605] req@ffff8800a934e680 x1818247334489088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018605.664862:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2872 00000100:00080000:0.0:1734018605.664864:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018605.664868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018605.664872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018605.664879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334489856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018605.664879:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334489088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018605.664890:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018605.664894:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018605.664899:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334489088:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018605.664906:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334489216:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018605.664923:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018605.664930:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018605/real 1734018605] req@ffff8800a934ce00 x1818247334489216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018605.664941:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334489216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018605.664949:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018605.664953:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018605.664956:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334489216:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018605.664962:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334489344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018605.664976:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018605.664982:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018605/real 1734018605] req@ffff8800a934d500 x1818247334489344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018605.664992:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334489344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018605.665000:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018605.665002:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018605.665006:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334489344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018605.665031:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334489472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018605.665044:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018605.665050:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018605/real 1734018605] req@ffff8800a934fb80 x1818247334489472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018605.665061:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334489472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018605.665068:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018605.665071:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018605.665074:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334489472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018605.665079:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334489600:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018605.665095:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018605.665101:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018605/real 1734018605] req@ffff8800a934f100 x1818247334489600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018605.665111:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334489600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018605.665118:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018605.665121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018605.665124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334489600:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018605.665129:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334489728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018605.665145:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018605.665151:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018605/real 1734018605] req@ffff8800a934e300 x1818247334489728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018605.665161:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334489728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018605.665168:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018605.665171:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018605.665174:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334489728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018605.665179:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334489856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018605.665191:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018605.665197:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018605/real 1734018605] req@ffff8800a934dc00 x1818247334489856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018605.665265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334489856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018605.665275:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018605.665278:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018605.665281:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334489856:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018606.554787:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018607.556770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018608.558733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018609.560721:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018610.562761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018610.672604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018610.672615:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018610.672619:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018610.672631:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2877 00000100:00080000:0.0:1734018610.672640:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018610.672648:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018610.672686:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334489984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018610.672718:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018610.672722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018610.672724:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018610.672729:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2877 00000100:00080000:0.0:1734018610.672732:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018610.672736:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018610.672746:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018610.672757:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334490112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018610.672768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018610.672771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018610.672773:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018610.672777:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2877 00000100:00080000:0.0:1734018610.672779:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018610.672783:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018610.672787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018610.672796:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334490240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018610.672805:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018610.672808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018610.672810:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018610.672813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2877 00000100:00080000:0.0:1734018610.672816:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018610.672819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018610.672819:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334489984:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018610.672823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018610.672830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334490368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018610.672839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018610.672842:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018610.672844:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018610.672847:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2877 00000100:00080000:0.0:1734018610.672850:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018610.672853:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018610.672858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018610.672865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334490496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018610.672873:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018610.672876:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018610.672877:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018610.672880:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2877 00000100:00080000:0.0:1734018610.672883:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734018610.672884:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018610.672887:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018610.672891:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018610.672899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334490624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018610.672906:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018610/real 1734018610] req@ffff8800a934c000 x1818247334489984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018610.672907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018610.672910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018610.672912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018610.672915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2877 00000100:00080000:0.0:1734018610.672918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018610.672921:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018610.672925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018610.672927:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334489984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018610.672933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334490752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018610.672938:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018610.672942:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018610.672947:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334489984:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018610.672954:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334490112:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018610.672971:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018610.672978:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018610/real 1734018610] req@ffff8800a934d180 x1818247334490112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018610.672989:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334490112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018610.672997:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018610.673001:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018610.673004:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334490112:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018610.673028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334490240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018610.673042:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018610.673049:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018610/real 1734018610] req@ffff8800a934c700 x1818247334490240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018610.673059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334490240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018610.673067:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018610.673069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018610.673072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334490240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018610.673078:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334490368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018610.673090:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018610.673096:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018610/real 1734018610] req@ffff8800a934ed80 x1818247334490368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018610.673106:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334490368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018610.673113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018610.673115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018610.673118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334490368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018610.673124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334490496:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018610.673182:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018610.673190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018610/real 1734018610] req@ffff8800a934ca80 x1818247334490496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018610.673201:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334490496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018610.673209:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018610.673212:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018610.673215:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334490496:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018610.673220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334490624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018610.673238:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018610.673244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018610/real 1734018610] req@ffff8800a934ea00 x1818247334490624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018610.673254:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334490624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018610.673262:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018610.673264:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018610.673267:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334490624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018610.673272:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334490752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018610.673285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018610.673291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018610/real 1734018610] req@ffff8800a934c380 x1818247334490752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018610.673307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334490752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018610.673315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018610.673317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018610.673320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334490752:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018611.564758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018612.566687:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018613.568755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018614.570754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018615.572797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018615.680591:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018615.680603:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018615.680608:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018615.680619:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2882 00000100:00080000:0.0:1734018615.680628:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018615.680636:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018615.680674:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334490880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018615.680709:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018615.680713:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018615.680715:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018615.680719:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2882 00000100:00080000:0.0:1734018615.680723:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018615.680726:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018615.680736:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018615.680748:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334491008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018615.680757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018615.680761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018615.680762:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018615.680766:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2882 00000100:00080000:0.0:1734018615.680769:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018615.680773:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018615.680779:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018615.680789:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334491136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018615.680798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018615.680801:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018615.680802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018615.680805:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2882 00000100:00080000:0.0:1734018615.680808:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018615.680811:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018615.680814:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334490880:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018615.680815:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018615.680823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334491264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018615.680832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018615.680835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018615.680837:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018615.680840:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2882 00000100:00080000:0.0:1734018615.680842:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018615.680845:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018615.680849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018615.680857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334491392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018615.680866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018615.680869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018615.680871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018615.680875:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2882 00000100:00080000:0.0:1734018615.680878:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734018615.680878:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018615.680881:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018615.680885:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018615.680893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334491520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018615.680900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734018615.680900:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018615/real 1734018615] req@ffff8800a934df80 x1818247334490880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018615.680903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018615.680905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018615.680908:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2882 00000100:00080000:0.0:1734018615.680911:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018615.680914:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018615.680918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018615.680920:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334490880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018615.680925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334491648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018615.680931:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018615.680934:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018615.680939:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334490880:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018615.680947:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334491008:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018615.680963:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018615.680970:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018615/real 1734018615] req@ffff8800a934d880 x1818247334491008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018615.680982:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334491008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018615.680989:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018615.680993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018615.680997:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334491008:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018615.681003:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334491136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018615.681016:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018615.681022:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018615/real 1734018615] req@ffff8800a934c380 x1818247334491136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018615.681032:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334491136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018615.681040:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018615.681042:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018615.681046:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334491136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018615.681111:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334491264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018615.681128:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018615.681135:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018615/real 1734018615] req@ffff8800a934ea00 x1818247334491264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018615.681146:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334491264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018615.681153:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018615.681156:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018615.681159:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334491264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018615.681164:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334491392:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018615.681180:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018615.681186:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018615/real 1734018615] req@ffff8800a934ca80 x1818247334491392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018615.681197:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334491392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018615.681205:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018615.681207:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018615.681210:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334491392:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018615.681215:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334491520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018615.681226:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018615.681232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018615/real 1734018615] req@ffff8800a934ed80 x1818247334491520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018615.681242:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334491520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018615.681249:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018615.681251:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018615.681254:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334491520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018615.681259:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334491648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018615.681272:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018615.681278:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018615/real 1734018615] req@ffff8800a934c700 x1818247334491648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018615.681295:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334491648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018615.681302:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018615.681304:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018615.681307:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334491648:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018616.574747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018617.576687:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018618.578793:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018619.580772:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018620.582732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018620.688616:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018620.688628:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018620.688633:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018620.688643:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2887 00000100:00080000:0.0:1734018620.688652:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018620.688660:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018620.688698:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334491776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018620.688739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018620.688744:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018620.688746:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018620.688750:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2887 00000100:00080000:0.0:1734018620.688753:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018620.688757:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018620.688767:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018620.688778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334491904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018620.688788:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018620.688791:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018620.688793:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018620.688797:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2887 00000100:00080000:0.0:1734018620.688799:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018620.688803:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018620.688808:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018620.688816:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334492032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018620.688825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018620.688828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018620.688830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018620.688833:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2887 00000100:00080000:0.0:1734018620.688835:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018620.688839:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018620.688843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018620.688848:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334491776:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018620.688851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334492160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018620.688861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018620.688863:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018620.688865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018620.688868:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2887 00000100:00080000:0.0:1734018620.688870:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018620.688874:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018620.688878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018620.688885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334492288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018620.688893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018620.688896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018620.688897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018620.688900:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2887 00000100:00080000:0.0:1734018620.688903:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018620.688906:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018620.688910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018620.688913:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018620.688920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334492416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018620.688928:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018620.688932:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018620.688933:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018620.688936:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018620/real 1734018620] req@ffff8800a934d180 x1818247334491776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018620.688937:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2887 00000100:00080000:0.0:1734018620.688940:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018620.688943:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018620.688947:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018620.688954:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334492544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018620.688956:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334491776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018620.688997:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018620.689001:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018620.689006:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334491776:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018620.689015:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334491904:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018620.689035:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018620.689043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018620/real 1734018620] req@ffff8800a934c000 x1818247334491904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018620.689055:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334491904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018620.689063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018620.689067:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018620.689070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334491904:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018620.689076:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334492032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018620.689090:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018620.689096:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018620/real 1734018620] req@ffff8800a934dc00 x1818247334492032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018620.689106:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334492032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018620.689114:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018620.689116:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018620.689120:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334492032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018620.689143:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334492160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018620.689156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018620.689162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018620/real 1734018620] req@ffff8800a934e300 x1818247334492160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018620.689173:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334492160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018620.689181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018620.689183:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018620.689186:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334492160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018620.689191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334492288:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018620.689206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018620.689213:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018620/real 1734018620] req@ffff8800a934f100 x1818247334492288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018620.689223:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334492288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018620.689230:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018620.689232:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018620.689235:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334492288:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018620.689240:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334492416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018620.689256:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018620.689262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018620/real 1734018620] req@ffff8800a934fb80 x1818247334492416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018620.689272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334492416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018620.689280:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018620.689282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018620.689285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334492416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018620.689290:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334492544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018620.689302:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018620.689308:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018620/real 1734018620] req@ffff8800a934d500 x1818247334492544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018620.689324:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334492544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018620.689331:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018620.689333:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018620.689336:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334492544:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018621.584743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018622.586715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018623.588748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018624.590729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018625.592771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018625.696609:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018625.696621:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018625.696625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018625.696636:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2892 00000100:00080000:0.0:1734018625.696645:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018625.696652:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018625.696689:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334492672/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018625.696724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018625.696728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018625.696730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018625.696735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2892 00000100:00080000:0.0:1734018625.696738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018625.696742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018625.696752:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018625.696764:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334492800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018625.696775:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018625.696779:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018625.696780:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018625.696784:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2892 00000100:00080000:0.0:1734018625.696787:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018625.696790:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018625.696795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018625.696805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334492928/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018625.696815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018625.696818:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018625.696819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018625.696823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2892 00000100:00080000:0.0:1734018625.696825:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018625.696826:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334492672:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018625.696828:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018625.696833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018625.696840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334493056/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018625.696849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018625.696852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018625.696854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018625.696857:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2892 00000100:00080000:0.0:1734018625.696860:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018625.696864:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018625.696868:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018625.696876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334493184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018625.696912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018625.696915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1734018625.696916:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018625.696917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018625.696920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2892 00000100:00080000:0.0:1734018625.696925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018625.696928:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018625.696933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018625.696940:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018625/real 1734018625] req@ffff8800a934ce00 x1818247334492672/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018625.696942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334493312/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018625.696950:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018625.696953:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018625.696955:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018625.696958:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2892 00000100:00100000:2.0:1734018625.696960:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334492672/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018625.696961:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018625.696964:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018625.696968:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734018625.696971:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018625.696974:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018625.696976:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334493440/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018625.696979:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334492672:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018625.696987:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334492800:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018625.697005:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018625.697013:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018625/real 1734018625] req@ffff8800a934e680 x1818247334492800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018625.697024:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334492800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018625.697032:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018625.697036:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018625.697039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334492800:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018625.697062:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334492928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018625.697076:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018625.697082:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018625/real 1734018625] req@ffff8800a934d500 x1818247334492928/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018625.697093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334492928/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018625.697100:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018625.697103:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018625.697106:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334492928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018625.697111:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334493056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018625.697123:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018625.697129:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018625/real 1734018625] req@ffff8800a934fb80 x1818247334493056/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018625.697139:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334493056/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018625.697146:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018625.697148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018625.697151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334493056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018625.697157:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334493184:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018625.697173:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018625.697180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018625/real 1734018625] req@ffff8800a934f100 x1818247334493184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018625.697190:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334493184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018625.697197:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018625.697199:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018625.697202:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334493184:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018625.697207:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334493312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018625.697222:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018625.697228:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018625/real 1734018625] req@ffff8800a934e300 x1818247334493312/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018625.697237:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334493312/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018625.697245:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018625.697247:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018625.697250:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334493312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018625.697255:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334493440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018625.697267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018625.697273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018625/real 1734018625] req@ffff8800a934dc00 x1818247334493440/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018625.697295:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334493440/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018625.697302:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018625.697304:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018625.697307:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334493440:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018626.594772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018627.596758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018628.598704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018629.600750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018630.602752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018630.704600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018630.704611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018630.704616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018630.704627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2897 00000100:00080000:0.0:1734018630.704636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018630.704644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018630.704680:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334493568/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018630.704713:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018630.704718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018630.704720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018630.704724:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2897 00000100:00080000:0.0:1734018630.704728:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018630.704732:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018630.704742:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018630.704753:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334493696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018630.704762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018630.704765:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018630.704767:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018630.704771:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2897 00000100:00080000:0.0:1734018630.704774:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018630.704778:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018630.704783:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018630.704821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334493824/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018630.704832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018630.704835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018630.704837:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018630.704841:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2897 00000100:00080000:0.0:1734018630.704844:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018630.704847:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018630.704851:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018630.704859:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334493568:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018630.704860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334493952/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018630.704870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018630.704873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018630.704875:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018630.704878:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2897 00000100:00080000:0.0:1734018630.704880:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018630.704883:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018630.704887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018630.704895:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334494080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018630.704903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018630.704905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018630.704908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018630.704911:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2897 00000100:00080000:0.0:1734018630.704914:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018630.704918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018630.704922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018630.704924:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018630.704931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334494208/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018630.704939:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018630.704942:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018630.704943:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018630.704947:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2897 00000100:00100000:2.0:1734018630.704947:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018630/real 1734018630] req@ffff8800a934c000 x1818247334493568/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018630.704949:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018630.704952:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018630.704956:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018630.704964:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334494336/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018630.704968:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334493568/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018630.704978:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018630.704981:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018630.704986:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334493568:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018630.704993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334493696:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018630.705010:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018630.705017:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018630/real 1734018630] req@ffff8800a934d180 x1818247334493696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018630.705028:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334493696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018630.705035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018630.705039:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018630.705043:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334493696:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018630.705049:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334493824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018630.705062:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018630.705068:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018630/real 1734018630] req@ffff8800a934c700 x1818247334493824/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018630.705078:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334493824/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018630.705086:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018630.705088:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018630.705092:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334493824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018630.705115:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334493952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018630.705128:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018630.705135:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018630/real 1734018630] req@ffff8800a934ed80 x1818247334493952/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018630.705145:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334493952/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018630.705153:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018630.705155:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018630.705158:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334493952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018630.705163:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334494080:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018630.705179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018630.705185:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018630/real 1734018630] req@ffff8800a934ca80 x1818247334494080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018630.705195:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334494080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018630.705202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018630.705204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018630.705207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334494080:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018630.705212:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334494208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018630.705224:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018630.705230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018630/real 1734018630] req@ffff8800a934ea00 x1818247334494208/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018630.705240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334494208/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018630.705247:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018630.705249:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018630.705252:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334494208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018630.705257:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334494336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018630.705269:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018630.705275:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018630/real 1734018630] req@ffff8800a934c380 x1818247334494336/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018630.705292:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334494336/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018630.705300:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018630.705302:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018630.705305:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334494336:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018631.604745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018632.606748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018633.608820:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018634.610801:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018635.612804:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018635.712578:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018635.712595:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018635.712599:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018635.712610:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2902 00000100:00080000:0.0:1734018635.712619:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018635.712627:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018635.712664:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334494464/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018635.712705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018635.712748:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018635.712750:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018635.712755:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2902 00000100:00080000:0.0:1734018635.712758:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018635.712763:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018635.712778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018635.712791:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334494592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018635.712802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018635.712805:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018635.712807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018635.712811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2902 00000100:00080000:0.0:1734018635.712813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018635.712814:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334494464:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018635.712817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018635.712822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018635.712832:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334494720/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018635.712841:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018635.712844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018635.712845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018635.712849:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2902 00000100:00080000:0.0:1734018635.712851:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018635.712855:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018635.712860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018635.712867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334494848/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018635.712876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018635.712878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1734018635.712879:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018635.712880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018635.712883:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2902 00000100:00080000:0.0:1734018635.712886:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018635.712889:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018635.712893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018635.712900:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018635/real 1734018635] req@ffff8800a934d880 x1818247334494464/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018635.712901:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334494976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018635.712909:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018635.712912:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018635.712913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018635.712917:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2902 00000100:00080000:0.0:1734018635.712920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734018635.712920:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334494464/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018635.712923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018635.712927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734018635.712931:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1734018635.712934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334495104/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018635.712934:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018635.712939:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334494464:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018635.712942:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018635.712946:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018635.712948:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018635.712951:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2902 00000100:00080000:0.0:1734018635.712954:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018635.712957:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018635.712961:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018635.712962:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334494592:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018635.712968:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334495232/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734018635.712978:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018635.712986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018635/real 1734018635] req@ffff8800a934df80 x1818247334494592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018635.712997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334494592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018635.713005:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018635.713009:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018635.713013:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334494592:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018635.713020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334494720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018635.713033:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018635.713039:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018635/real 1734018635] req@ffff8800a934c380 x1818247334494720/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018635.713049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334494720/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018635.713058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018635.713060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018635.713064:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334494720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018635.713070:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334494848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018635.713082:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018635.713088:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018635/real 1734018635] req@ffff8800a934ea00 x1818247334494848/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018635.713097:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334494848/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018635.713105:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018635.713107:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018635.713111:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334494848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018635.713116:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334494976:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018635.713132:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018635.713139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018635/real 1734018635] req@ffff8800a934ca80 x1818247334494976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018635.713149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334494976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018635.713156:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018635.713158:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018635.713161:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334494976:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018635.713167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334495104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018635.713179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018635.713185:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018635/real 1734018635] req@ffff8800a934ed80 x1818247334495104/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018635.713195:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334495104/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018635.713202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018635.713204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018635.713207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334495104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018635.713224:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334495232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018635.713237:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018635.713243:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018635/real 1734018635] req@ffff8800a934c700 x1818247334495232/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018635.713260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334495232/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018635.713268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018635.713270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018635.713273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334495232:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018636.614766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018637.616764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018638.618785:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018639.620797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018640.622741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018640.720712:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018640.720722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018640.720727:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018640.720738:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2907 00000100:00080000:2.0:1734018640.720747:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018640.720755:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018640.720795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334495360/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018640.720833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018640.720837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018640.720839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018640.720843:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2907 00000100:00080000:2.0:1734018640.720846:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018640.720850:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018640.720860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018640.720872:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334495488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018640.720888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018640.720892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018640.720893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018640.720897:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2907 00000100:00080000:2.0:1734018640.720900:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018640.720903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018640.720908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018640.720916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334495616/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018640.720925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018640.720928:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018640.720929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018640.720930:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334495360:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018640.720933:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2907 00000100:00080000:2.0:1734018640.720935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018640.720939:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018640.720943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018640.720950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334495744/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018640.720958:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018640.720962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018640.720963:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018640.720967:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2907 00000100:00080000:2.0:1734018640.720970:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018640.720973:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018640.720977:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018640.720985:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334495872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018640.720992:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018640.720995:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018640.720997:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018640.721000:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2907 00000100:00080000:2.0:1734018640.721003:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018640.721006:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018640.721010:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018640.721018:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334496000/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018640.721023:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334495488:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734018640.721025:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018640.721028:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018640.721030:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018640.721034:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2907 00000100:00080000:2.0:1734018640.721036:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018640.721039:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018640.721043:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018640.721050:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334495616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018640.721051:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334496128/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018640.721066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334495744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018640.721080:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334495872:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018640.721095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334496000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018640.721112:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334496128:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018641.624773:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018642.626777:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018643.628755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018643.726794:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734018643.726850:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018643.726874:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018643.726924:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018643.726934:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018643.726941:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018643.726949:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018643.726956:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018643.726963:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734018643.727063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018640/real 1734018643] req@ffff8800a78a3b80 x1818247334496128/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018643.727100:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334496128/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018643.727112:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018643.727116:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018643.727122:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334496128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018643.727131:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018640/real 1734018643] req@ffff8800a78a1f80 x1818247334495616/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018643.727146:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334495616/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018643.727160:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018643.727164:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018643.727171:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334495616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018643.727182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018640/real 1734018643] req@ffff8800a78a0a80 x1818247334495744/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018643.727200:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334495744/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018643.727213:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018643.727217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018643.727222:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334495744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018643.727232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018640/real 1734018643] req@ffff8800a78a2680 x1818247334495872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018643.727250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334495872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018643.727263:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018643.727267:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018643.727272:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334495872:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018643.727283:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018640/real 1734018643] req@ffff8800a78a1880 x1818247334496000/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018643.727301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334496000/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018643.727314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018643.727318:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018643.727336:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334496000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018643.727347:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018640/real 1734018643] req@ffff8800a78a0700 x1818247334495360/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018643.727376:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334495360/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018643.727400:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018643.727405:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018643.727410:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334495360:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018643.727421:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018640/real 1734018643] req@ffff8800a78a3480 x1818247334495488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018643.727455:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334495488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018643.727525:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018643.727530:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018643.727537:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334495488:192.168.201.116@tcp:38:kworker.0 00010000:00010000:3.0:1734018644.630754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018645.632779:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018645.728612:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018645.728623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018645.728628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018645.728641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2912 00000100:00080000:2.0:1734018645.728651:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018645.728661:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018645.728714:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334496256/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018645.728752:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018645.728757:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018645.728759:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018645.728764:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2912 00000100:00080000:2.0:1734018645.728767:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018645.728771:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018645.728782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018645.728794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334496384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018645.728804:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018645.728808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018645.728810:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018645.728814:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2912 00000100:00080000:2.0:1734018645.728817:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018645.728821:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018645.728827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018645.728837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334496512/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018645.728847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018645.728851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018645.728852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018645.728856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2912 00000100:00080000:2.0:1734018645.728858:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018645.728862:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018645.728866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018645.728874:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334496640/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018645.728884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018645.728887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018645.728889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018645.728892:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334496256:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018645.728893:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2912 00000100:00080000:2.0:1734018645.728895:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018645.728899:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018645.728903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018645.728911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334496768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018645.728920:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018645.728923:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018645.728926:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018645.728930:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2912 00000100:00080000:2.0:1734018645.728932:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018645.728936:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018645.728940:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018645.728948:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334496896/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018645.728956:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018645.728959:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018645.728961:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018645.728964:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2912 00000800:00000100:1.0:1734018645.728966:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018645.728968:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018645.728971:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018645.728975:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018645.728983:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334497024/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018645.728987:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018645/real 1734018645] req@ffff8800a78a3480 x1818247334496256/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.729007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334496256/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.729019:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.729022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018645.729027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334496256:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018645.729035:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334496384:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018645.729052:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018645.729060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018645/real 1734018645] req@ffff8800a78a0700 x1818247334496384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.729071:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334496384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.729079:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.729082:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018645.729086:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334496384:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018645.729092:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334496512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018645.729106:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018645.729113:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018645/real 1734018645] req@ffff8800a78a1880 x1818247334496512/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.729123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334496512/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.729131:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.729133:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018645.729137:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334496512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018645.729162:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334496640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018645.729176:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018645.729182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018645/real 1734018645] req@ffff8800a78a2680 x1818247334496640/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.729192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334496640/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.729207:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.729210:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018645.729213:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334496640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018645.729219:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334496768:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018645.729232:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018645.729238:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018645/real 1734018645] req@ffff8800a78a0a80 x1818247334496768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.729248:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334496768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.729255:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.729258:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018645.729261:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334496768:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018645.729267:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334496896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018645.729279:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018645.729285:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018645/real 1734018645] req@ffff8800a78a1f80 x1818247334496896/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.729295:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334496896/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.729302:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.729304:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018645.729307:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334496896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018645.729314:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334497024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018645.729327:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018645.729333:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018645/real 1734018645] req@ffff8800a78a3b80 x1818247334497024/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.729343:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334497024/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.729350:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.729352:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018645.729355:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334497024:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018646.634778:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018647.636741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018648.638751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018649.640769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018650.642750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018650.736612:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018650.736624:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018650.736629:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018650.736640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2917 00000100:00080000:2.0:1734018650.736648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018650.736656:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018650.736694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334497152/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018650.736737:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018650.736741:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018650.736743:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018650.736748:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2917 00000100:00080000:2.0:1734018650.736751:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018650.736755:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018650.736765:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018650.736777:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334497280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018650.736786:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018650.736789:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018650.736790:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018650.736795:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2917 00000100:00080000:2.0:1734018650.736798:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018650.736802:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018650.736806:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018650.736815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334497408/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018650.736823:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018650.736827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018650.736828:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018650.736831:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2917 00000100:00080000:2.0:1734018650.736834:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018650.736837:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018650.736842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018650.736850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334497536/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018650.736852:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334497152:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018650.736859:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018650.736862:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018650.736863:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018650.736866:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2917 00000100:00080000:2.0:1734018650.736869:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018650.736872:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018650.736876:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018650.736883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334497664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018650.736891:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018650.736894:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018650.736895:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018650.736898:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2917 00000100:00080000:2.0:1734018650.736901:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018650.736905:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018650.736908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018650.736916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334497792/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018650.736918:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018650.736925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018650.736928:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018650.736930:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018650.736933:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2917 00000100:00080000:2.0:1734018650.736936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018650.736939:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018650/real 1734018650] req@ffff8800a78a3b80 x1818247334497152/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018650.736939:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018650.736943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018650.736951:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334497920/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018650.736959:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334497152/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018650.736969:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018650.736972:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018650.736977:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334497152:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018650.736984:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334497280:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018650.737001:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018650.737008:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018650/real 1734018650] req@ffff8800a78a1f80 x1818247334497280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018650.737027:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334497280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018650.737035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018650.737038:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018650.737041:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334497280:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018650.737047:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334497408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018650.737060:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018650.737066:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018650/real 1734018650] req@ffff8800a78a0a80 x1818247334497408/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018650.737076:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334497408/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018650.737084:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018650.737086:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018650.737089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334497408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018650.737113:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334497536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018650.737126:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018650.737132:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018650/real 1734018650] req@ffff8800a78a2680 x1818247334497536/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018650.737143:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334497536/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018650.737150:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018650.737153:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018650.737156:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334497536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018650.737162:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334497664:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018650.737174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018650.737180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018650/real 1734018650] req@ffff8800a78a1880 x1818247334497664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018650.737190:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334497664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018650.737197:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018650.737199:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018650.737202:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334497664:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018650.737207:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334497792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018650.737219:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018650.737224:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018650/real 1734018650] req@ffff8800a78a0700 x1818247334497792/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018650.737234:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334497792/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018650.737241:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018650.737243:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018650.737246:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334497792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018650.737251:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334497920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018650.737267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018650.737273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018650/real 1734018650] req@ffff8800a78a3480 x1818247334497920/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018650.737282:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334497920/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018650.737289:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018650.737291:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018650.737294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334497920:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018651.644731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018652.646734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018653.648752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018654.650747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018655.652738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018655.744592:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018655.744602:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018655.744606:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018655.744616:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2922 00000100:00080000:2.0:1734018655.744625:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018655.744633:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018655.744670:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334498048/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018655.744705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018655.744709:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018655.744711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018655.744716:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2922 00000100:00080000:2.0:1734018655.744719:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018655.744723:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018655.744733:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018655.744744:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334498176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018655.744755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018655.744759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018655.744760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018655.744764:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2922 00000100:00080000:2.0:1734018655.744767:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018655.744770:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018655.744775:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018655.744783:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334498304/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018655.744792:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018655.744795:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018655.744797:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018655.744800:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2922 00000100:00080000:2.0:1734018655.744802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018655.744805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018655.744809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018655.744814:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334498048:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018655.744817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334498432/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018655.744826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018655.744829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018655.744830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018655.744834:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2922 00000100:00080000:2.0:1734018655.744837:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018655.744840:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018655.744844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018655.744852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334498560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018655.744860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018655.744863:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018655.744864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018655.744867:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2922 00000100:00080000:2.0:1734018655.744870:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018655.744873:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018655.744877:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018655.744879:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018655.744885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334498688/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018655.744893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018655.744896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018655.744897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018655.744901:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018655/real 1734018655] req@ffff8800a78a3480 x1818247334498048/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018655.744901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2922 00000100:00080000:2.0:1734018655.744904:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018655.744907:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018655.744911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018655.744918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334498816/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018655.744921:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334498048/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018655.744931:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018655.744935:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018655.744939:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334498048:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018655.744947:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334498176:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018655.744963:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018655.744970:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018655/real 1734018655] req@ffff8800a78a0700 x1818247334498176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018655.744982:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334498176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018655.744991:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018655.744994:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018655.744998:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334498176:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018655.745004:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334498304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018655.745017:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018655.745023:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018655/real 1734018655] req@ffff8800a78a1880 x1818247334498304/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018655.745033:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334498304/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018655.745041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018655.745043:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018655.745046:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334498304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018655.745071:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334498432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018655.745084:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018655.745091:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018655/real 1734018655] req@ffff8800a78a2680 x1818247334498432/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018655.745101:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334498432/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018655.745109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018655.745111:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018655.745114:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334498432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018655.745120:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334498560:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018655.745132:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018655.745138:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018655/real 1734018655] req@ffff8800a78a0a80 x1818247334498560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018655.745148:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334498560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018655.745155:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018655.745157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018655.745160:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334498560:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018655.745165:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334498688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018655.745177:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018655.745182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018655/real 1734018655] req@ffff8800a78a1f80 x1818247334498688/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018655.745192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334498688/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018655.745200:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018655.745202:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018655.745205:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334498688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018655.745210:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334498816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018655.745222:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018655.745228:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018655/real 1734018655] req@ffff8800a78a3b80 x1818247334498816/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018655.745239:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334498816/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018655.745247:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018655.745249:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018655.745252:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334498816:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018656.654744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018657.656728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018658.658761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018659.660761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018660.662707:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018660.752613:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018660.752622:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018660.752627:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018660.752638:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2927 00000100:00080000:2.0:1734018660.752646:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018660.752654:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018660.752693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334498944/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018660.752728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018660.752731:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018660.752733:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018660.752738:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2927 00000100:00080000:2.0:1734018660.752741:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018660.752745:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018660.752755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018660.752767:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334499072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018660.752777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018660.752780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018660.752782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018660.752786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2927 00000100:00080000:2.0:1734018660.752789:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018660.752793:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018660.752799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018660.752808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334499200/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018660.752817:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018660.752820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018660.752821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018660.752825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2927 00000100:00080000:2.0:1734018660.752827:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018660.752830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018660.752834:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334498944:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734018660.752834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018660.752842:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334499328/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018660.752851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018660.752854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018660.752856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018660.752859:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2927 00000100:00080000:2.0:1734018660.752861:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018660.752864:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018660.752868:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018660.752876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334499456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018660.752884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018660.752887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018660.752889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018660.752893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2927 00000100:00080000:2.0:1734018660.752896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018660.752900:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018660.752900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018660.752904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018660.752911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334499584/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018660.752919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018660.752922:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018660/real 1734018660] req@ffff8800a78a3b80 x1818247334498944/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018660.752922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018660.752923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018660.752927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2927 00000100:00080000:2.0:1734018660.752930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018660.752933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018660.752936:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018660.752943:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334498944/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018660.752944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334499712/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018660.752954:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018660.752957:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018660.752962:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334498944:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018660.752969:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334499072:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018660.752985:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018660.752992:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018660/real 1734018660] req@ffff8800a78a1f80 x1818247334499072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018660.753003:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334499072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018660.753011:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018660.753014:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018660.753018:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334499072:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018660.753024:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334499200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018660.753043:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018660.753049:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018660/real 1734018660] req@ffff8800a78a0a80 x1818247334499200/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018660.753059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334499200/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018660.753066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018660.753069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018660.753072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334499200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018660.753096:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334499328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018660.753109:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018660.753115:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018660/real 1734018660] req@ffff8800a78a2680 x1818247334499328/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018660.753125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334499328/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018660.753133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018660.753135:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018660.753138:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334499328:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018660.753143:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334499456:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018660.753159:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018660.753165:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018660/real 1734018660] req@ffff8800a78a1880 x1818247334499456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018660.753176:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334499456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018660.753183:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018660.753185:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018660.753188:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334499456:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018660.753193:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334499584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018660.753205:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018660.753210:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018660/real 1734018660] req@ffff8800a78a0700 x1818247334499584/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018660.753220:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334499584/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018660.753227:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018660.753229:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018660.753232:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334499584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018660.753237:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334499712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018660.753249:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018660.753255:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018660/real 1734018660] req@ffff8800a78a3480 x1818247334499712/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018660.753271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334499712/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018660.753278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018660.753280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018660.753313:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334499712:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018661.664737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018662.666697:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018663.668725:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018664.670705:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018665.672743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018665.760577:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018665.760586:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018665.760590:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018665.760601:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2932 00000100:00080000:2.0:1734018665.760610:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018665.760618:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018665.760657:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334499840/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018665.760700:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018665.760704:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018665.760706:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018665.760710:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2932 00000100:00080000:2.0:1734018665.760714:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018665.760718:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018665.760728:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018665.760739:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334499968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018665.760748:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018665.760752:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018665.760753:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018665.760757:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2932 00000100:00080000:2.0:1734018665.760760:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018665.760764:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018665.760769:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018665.760778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334500096/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018665.760786:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018665.760789:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018665.760791:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018665.760794:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2932 00000100:00080000:2.0:1734018665.760796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018665.760800:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018665.760804:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018665.760812:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334500224/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018665.760813:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334499840:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018665.760821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018665.760824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018665.760825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018665.760828:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2932 00000100:00080000:2.0:1734018665.760831:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018665.760834:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018665.760838:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018665.760845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334500352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018665.760853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018665.760855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018665.760857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018665.760860:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2932 00000100:00080000:2.0:1734018665.760872:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018665.760876:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018665.760877:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734018665.760880:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018665.760888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334500480/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018665.760896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018665.760898:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018665/real 1734018665] req@ffff8800a78a3480 x1818247334499840/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018665.760900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018665.760902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018665.760906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2932 00000100:00080000:2.0:1734018665.760910:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018665.760913:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018665.760917:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018665.760918:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334499840/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018665.760925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334500608/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018665.760928:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018665.760932:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018665.760936:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334499840:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018665.760944:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334499968:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018665.760960:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018665.760967:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018665/real 1734018665] req@ffff8800a78a0700 x1818247334499968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018665.760978:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334499968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018665.760986:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018665.760989:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018665.760993:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334499968:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018665.760999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334500096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018665.761013:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018665.761019:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018665/real 1734018665] req@ffff8800a78a1880 x1818247334500096/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018665.761029:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334500096/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018665.761036:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018665.761039:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018665.761042:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334500096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018665.761066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334500224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018665.761079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018665.761085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018665/real 1734018665] req@ffff8800a78a2680 x1818247334500224/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018665.761095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334500224/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018665.761103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018665.761105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018665.761108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334500224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018665.761113:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334500352:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018665.761129:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018665.761135:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018665/real 1734018665] req@ffff8800a78a0a80 x1818247334500352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018665.761146:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334500352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018665.761153:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018665.761155:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018665.761158:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334500352:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018665.761163:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334500480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018665.761174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018665.761180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018665/real 1734018665] req@ffff8800a78a1f80 x1818247334500480/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018665.761190:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334500480/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018665.761227:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018665.761229:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018665.761233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334500480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018665.761239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334500608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018665.761255:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018665.761262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018665/real 1734018665] req@ffff8800a78a3b80 x1818247334500608/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018665.761279:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334500608/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018665.761287:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018665.761289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018665.761292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334500608:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018666.674729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018667.676733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018668.678695:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018669.680762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018670.682746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018670.768574:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018670.768588:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018670.768593:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018670.768603:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2937 00000100:00080000:2.0:1734018670.768613:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018670.768620:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018670.768659:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334500736/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018670.768695:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018670.768699:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018670.768701:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018670.768706:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2937 00000100:00080000:2.0:1734018670.768709:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018670.768713:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018670.768723:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018670.768734:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334500864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018670.768745:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018670.768748:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018670.768750:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018670.768754:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2937 00000100:00080000:2.0:1734018670.768757:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018670.768760:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018670.768765:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018670.768773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334500992/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018670.768782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018670.768785:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018670.768786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018670.768790:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2937 00000100:00080000:2.0:1734018670.768792:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018670.768795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018670.768800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018670.768808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334501120/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018670.768811:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334500736:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018670.768817:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018670.768820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018670.768821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018670.768825:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2937 00000100:00080000:2.0:1734018670.768828:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018670.768831:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018670.768835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018670.768843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334501248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018670.768851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018670.768854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018670.768855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018670.768858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2937 00000100:00080000:2.0:1734018670.768862:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018670.768865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018670.768869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018670.768877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334501376/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018670.768878:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018670.768884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018670.768887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018670.768889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018670.768892:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2937 00000100:00080000:2.0:1734018670.768895:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018670.768899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018670.768901:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018670/real 1734018670] req@ffff8800a78a3b80 x1818247334500736/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018670.768903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018670.768910:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334501504/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018670.768920:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334500736/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018670.768932:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018670.768935:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018670.768940:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334500736:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018670.768947:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334500864:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018670.768964:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018670.768970:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018670/real 1734018670] req@ffff8800a78a1f80 x1818247334500864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018670.768982:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334500864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018670.768990:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018670.768993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018670.768997:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334500864:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018670.769003:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334500992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018670.769017:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018670.769023:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018670/real 1734018670] req@ffff8800a78a0a80 x1818247334500992/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018670.769033:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334500992/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018670.769041:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018670.769043:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018670.769047:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334500992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018670.769071:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334501120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018670.769084:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018670.769090:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018670/real 1734018670] req@ffff8800a78a2680 x1818247334501120/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018670.769101:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334501120/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018670.769108:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018670.769111:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018670.769158:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334501120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018670.769165:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334501248:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018670.769185:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018670.769192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018670/real 1734018670] req@ffff8800a78a1880 x1818247334501248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018670.769203:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334501248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018670.769211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018670.769213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018670.769216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334501248:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018670.769222:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334501376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018670.769234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018670.769240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018670/real 1734018670] req@ffff8800a78a0700 x1818247334501376/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018670.769250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334501376/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018670.769257:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018670.769259:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018670.769262:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334501376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018670.769267:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334501504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018670.769280:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018670.769286:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018670/real 1734018670] req@ffff8800a78a3480 x1818247334501504/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018670.769304:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334501504/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018670.769312:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018670.769314:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018670.769317:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334501504:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018671.684799:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018672.686741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018673.688761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018674.690762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018675.692716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018675.776599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018675.776609:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018675.776614:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018675.776624:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2942 00000100:00080000:2.0:1734018675.776633:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018675.776641:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018675.776680:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334501632/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018675.776715:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018675.776719:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018675.776721:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018675.776726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2942 00000100:00080000:2.0:1734018675.776729:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018675.776733:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018675.776743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018675.776755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334501760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018675.776764:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018675.776767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018675.776769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018675.776774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2942 00000100:00080000:2.0:1734018675.776776:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018675.776781:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018675.776787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018675.776797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334501888/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018675.776807:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018675.776810:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018675.776811:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018675.776815:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2942 00000100:00080000:2.0:1734018675.776818:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018675.776821:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334501632:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018675.776821:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018675.776825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018675.776833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334502016/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018675.776841:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018675.776844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018675.776846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018675.776849:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2942 00000100:00080000:2.0:1734018675.776852:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018675.776855:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018675.776859:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018675.776866:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334502144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018675.776874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018675.776877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018675.776879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018675.776882:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2942 00000100:00080000:2.0:1734018675.776885:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018675.776887:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018675.776889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018675.776893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018675.776900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334502272/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018675.776908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018675.776909:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018675/real 1734018675] req@ffff8800a78a3480 x1818247334501632/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018675.776911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018675.776912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018675.776916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2942 00000100:00080000:2.0:1734018675.776918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018675.776921:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018675.776925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018675.776928:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334501632/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018675.776932:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334502400/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018675.776940:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018675.776943:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018675.776948:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334501632:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018675.776955:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334501760:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018675.776972:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018675.776979:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018675/real 1734018675] req@ffff8800a78a0700 x1818247334501760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018675.776991:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334501760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018675.776999:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018675.777002:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018675.777006:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334501760:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018675.777069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334501888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018675.777087:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018675.777095:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018675/real 1734018675] req@ffff8800a78a1880 x1818247334501888/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018675.777106:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334501888/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018675.777113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018675.777116:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018675.777119:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334501888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018675.777125:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334502016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018675.777137:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018675.777143:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018675/real 1734018675] req@ffff8800a78a2680 x1818247334502016/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018675.777152:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334502016/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018675.777160:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018675.777163:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018675.777166:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334502016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018675.777171:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334502144:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018675.777188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018675.777194:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018675/real 1734018675] req@ffff8800a78a0a80 x1818247334502144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018675.777204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334502144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018675.777211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018675.777213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018675.777216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334502144:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018675.777221:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334502272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018675.777234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018675.777239:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018675/real 1734018675] req@ffff8800a78a1f80 x1818247334502272/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018675.777249:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334502272/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018675.777256:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018675.777258:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018675.777262:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334502272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018675.777267:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334502400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018675.777279:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018675.777285:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018675/real 1734018675] req@ffff8800a78a3b80 x1818247334502400/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018675.777302:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334502400/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018675.777309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018675.777311:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018675.777314:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334502400:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018676.694783:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018677.696789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018678.698747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018679.700730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018680.702731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018680.784575:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018680.784584:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018680.784589:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018680.784600:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2947 00000100:00080000:2.0:1734018680.784609:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018680.784616:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018680.784655:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334502528/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018680.784698:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018680.784702:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018680.784704:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018680.784709:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2947 00000100:00080000:2.0:1734018680.784712:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018680.784716:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018680.784726:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018680.784738:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334502656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018680.784747:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018680.784750:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018680.784752:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018680.784756:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2947 00000100:00080000:2.0:1734018680.784758:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018680.784763:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018680.784767:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018680.784776:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334502784/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018680.784784:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018680.784787:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018680.784789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018680.784792:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2947 00000100:00080000:2.0:1734018680.784794:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018680.784798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018680.784803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018680.784806:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334502528:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018680.784811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334502912/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018680.784820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018680.784823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018680.784824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018680.784827:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2947 00000100:00080000:2.0:1734018680.784830:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018680.784833:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018680.784837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018680.784844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334503040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018680.784852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018680.784855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018680.784856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018680.784859:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2947 00000100:00080000:2.0:1734018680.784862:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018680.784865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018680.784869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018680.784871:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018680.784877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334503168/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018680.784885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018680.784888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018680.784890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018680.784891:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018680/real 1734018680] req@ffff8800a78a3b80 x1818247334502528/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018680.784894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2947 00000100:00080000:2.0:1734018680.784897:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018680.784900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018680.784904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018680.784911:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334502528/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018680.784911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334503296/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018680.784921:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018680.784925:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018680.784930:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334502528:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018680.784937:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334502656:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018680.784995:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018680.785003:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018680/real 1734018680] req@ffff8800a78a1f80 x1818247334502656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018680.785015:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334502656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018680.785024:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018680.785027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018680.785031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334502656:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018680.785037:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334502784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018680.785051:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018680.785058:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018680/real 1734018680] req@ffff8800a78a0a80 x1818247334502784/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018680.785068:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334502784/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018680.785075:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018680.785078:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018680.785081:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334502784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018680.785106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334502912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018680.785119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018680.785125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018680/real 1734018680] req@ffff8800a78a2680 x1818247334502912/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018680.785135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334502912/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018680.785143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018680.785145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018680.785148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334502912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018680.785153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334503040:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018680.785169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018680.785176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018680/real 1734018680] req@ffff8800a78a1880 x1818247334503040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018680.785186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334503040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018680.785193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018680.785196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018680.785199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334503040:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018680.785204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334503168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018680.785215:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018680.785221:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018680/real 1734018680] req@ffff8800a78a0700 x1818247334503168/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018680.785231:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334503168/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018680.785238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018680.785240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018680.785243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334503168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018680.785248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334503296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018680.785261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018680.785267:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018680/real 1734018680] req@ffff8800a78a3480 x1818247334503296/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018680.785283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334503296/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018680.785290:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018680.785292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018680.785295:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334503296:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018681.704727:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018682.706763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018683.708794:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018684.710751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018685.712705:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018685.792595:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018685.792604:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018685.792608:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018685.792619:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2952 00000100:00080000:2.0:1734018685.792628:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018685.792636:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018685.792674:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334503424/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018685.792708:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018685.792712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018685.792714:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018685.792718:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2952 00000100:00080000:2.0:1734018685.792722:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018685.792726:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018685.792737:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018685.792748:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334503552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018685.792759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018685.792762:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018685.792764:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018685.792768:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2952 00000100:00080000:2.0:1734018685.792771:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018685.792774:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018685.792779:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018685.792787:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334503680/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018685.792796:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018685.792799:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018685.792801:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018685.792804:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2952 00000100:00080000:2.0:1734018685.792806:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018685.792810:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018685.792814:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334503424:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734018685.792814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018685.792822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334503808/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018685.792831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018685.792834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018685.792835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018685.792838:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2952 00000100:00080000:2.0:1734018685.792842:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018685.792845:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018685.792849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018685.792890:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334503936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018685.792900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018685.792903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018685.792905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018685.792909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2952 00000100:00080000:2.0:1734018685.792912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018685.792916:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018685.792916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018685.792920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018685.792929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334504064/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018685.792938:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018685.792939:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018685/real 1734018685] req@ffff8800a78a3480 x1818247334503424/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018685.792941:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018685.792943:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018685.792946:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2952 00000100:00080000:2.0:1734018685.792949:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018685.792952:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018685.792956:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018685.792959:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334503424/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018685.792964:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334504192/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018685.792969:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018685.792972:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018685.792977:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334503424:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018685.792985:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334503552:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018685.793002:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018685.793009:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018685/real 1734018685] req@ffff8800a78a0700 x1818247334503552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018685.793020:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334503552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018685.793028:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018685.793032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018685.793035:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334503552:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018685.793042:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334503680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018685.793057:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018685.793063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018685/real 1734018685] req@ffff8800a78a1880 x1818247334503680/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018685.793073:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334503680/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018685.793081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018685.793083:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018685.793087:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334503680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018685.793111:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334503808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018685.793124:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018685.793130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018685/real 1734018685] req@ffff8800a78a2680 x1818247334503808/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018685.793140:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334503808/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018685.793148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018685.793150:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018685.793153:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334503808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018685.793159:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334503936:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018685.793174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018685.793180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018685/real 1734018685] req@ffff8800a78a0a80 x1818247334503936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018685.793190:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334503936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018685.793197:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018685.793200:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018685.793203:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334503936:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018685.793208:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334504064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018685.793219:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018685.793225:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018685/real 1734018685] req@ffff8800a78a1f80 x1818247334504064/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018685.793235:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334504064/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018685.793242:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018685.793244:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018685.793247:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334504064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018685.793253:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334504192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018685.793265:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018685.793271:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018685/real 1734018685] req@ffff8800a78a3b80 x1818247334504192/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018685.793288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334504192/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018685.793294:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018685.793297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018685.793300:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334504192:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018686.714729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018687.716752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018688.718721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018689.720775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018690.722697:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018690.800587:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018690.800600:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018690.800606:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018690.800619:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2957 00000100:00080000:2.0:1734018690.800630:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018690.800640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018690.800693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334504320/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018690.800731:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018690.800736:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018690.800738:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018690.800743:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2957 00000100:00080000:2.0:1734018690.800746:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018690.800750:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018690.800761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018690.800812:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334504448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018690.800823:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018690.800827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018690.800829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018690.800833:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2957 00000100:00080000:2.0:1734018690.800836:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018690.800840:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018690.800847:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018690.800857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334504576/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018690.800867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018690.800871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018690.800873:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334504320:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018690.800873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018690.800877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2957 00000100:00080000:2.0:1734018690.800879:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018690.800882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018690.800887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018690.800895:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334504704/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018690.800904:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018690.800907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018690.800908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018690.800912:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2957 00000100:00080000:2.0:1734018690.800914:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018690.800918:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018690.800922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018690.800931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334504832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018690.800939:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018690.800942:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018690.800944:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018690.800949:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2957 00000100:00080000:2.0:1734018690.800952:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018690.800956:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018690.800959:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734018690.800960:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018690.800969:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334504960/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018690.800976:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018690.800980:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018690.800981:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018690.800985:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2957 00000100:00100000:1.0:1734018690.800986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018690/real 1734018690] req@ffff8800a78a3b80 x1818247334504320/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018690.800988:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018690.800991:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018690.800995:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018690.801003:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334505088/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018690.801009:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334504320/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018690.801021:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018690.801024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018690.801029:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334504320:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018690.801037:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334504448:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018690.801055:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018690.801062:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018690/real 1734018690] req@ffff8800a78a1f80 x1818247334504448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018690.801074:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334504448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018690.801082:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018690.801086:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018690.801089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334504448:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018690.801118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334504576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018690.801134:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018690.801141:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018690/real 1734018690] req@ffff8800a78a0a80 x1818247334504576/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018690.801152:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334504576/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018690.801160:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018690.801162:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018690.801165:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334504576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018690.801171:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334504704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018690.801183:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018690.801189:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018690/real 1734018690] req@ffff8800a78a2680 x1818247334504704/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018690.801199:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334504704/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018690.801206:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018690.801208:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018690.801211:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334504704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018690.801218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334504832:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018690.801235:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018690.801241:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018690/real 1734018690] req@ffff8800a78a1880 x1818247334504832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018690.801251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334504832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018690.801258:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018690.801261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018690.801264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334504832:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018690.801269:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334504960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018690.801281:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018690.801287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018690/real 1734018690] req@ffff8800a78a0700 x1818247334504960/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018690.801297:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334504960/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018690.801304:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018690.801306:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018690.801309:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334504960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018690.801314:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334505088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018690.801327:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018690.801333:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018690/real 1734018690] req@ffff8800a78a3480 x1818247334505088/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018690.801349:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334505088/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018690.801356:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018690.801358:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018690.801361:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334505088:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018691.724733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018692.726803:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018693.728783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018694.730756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018695.732761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018695.808616:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018695.808629:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018695.808635:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018695.808649:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2962 00000100:00080000:2.0:1734018695.808660:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018695.808670:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018695.808762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334505216/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018695.808807:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018695.808812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018695.808814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018695.808818:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2962 00000100:00080000:2.0:1734018695.808823:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018695.808827:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018695.808839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018695.808852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334505344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018695.808862:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018695.808866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018695.808867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018695.808871:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2962 00000100:00080000:2.0:1734018695.808874:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018695.808878:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018695.808883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018695.808892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334505472/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018695.808901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018695.808904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018695.808906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018695.808909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2962 00000100:00080000:2.0:1734018695.808912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018695.808916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018695.808921:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018695.808928:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334505216:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018695.808930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334505600/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018695.808940:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018695.808943:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018695.808945:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018695.808948:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2962 00000100:00080000:2.0:1734018695.808951:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018695.808954:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018695.808959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018695.808966:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334505728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018695.808974:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018695.808977:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018695.808979:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018695.808983:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2962 00000100:00080000:2.0:1734018695.808986:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018695.808989:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018695.808993:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018695.809001:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334505856/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018695.809006:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018695.809009:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018695.809013:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018695.809015:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018695.809019:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2962 00000100:00080000:2.0:1734018695.809024:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018695.809027:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018695.809030:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018695/real 1734018695] req@ffff8800a78a3480 x1818247334505216/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018695.809031:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018695.809040:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334505984/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018695.809051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334505216/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018695.809063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018695.809066:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018695.809071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334505216:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018695.809080:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334505344:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018695.809098:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018695.809105:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018695/real 1734018695] req@ffff8800a78a0700 x1818247334505344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018695.809117:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334505344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018695.809125:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018695.809128:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018695.809132:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334505344:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018695.809139:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334505472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018695.809154:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018695.809160:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018695/real 1734018695] req@ffff8800a78a1880 x1818247334505472/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018695.809170:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334505472/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018695.809179:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018695.809181:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018695.809185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334505472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018695.809215:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334505600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018695.809229:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018695.809235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018695/real 1734018695] req@ffff8800a78a2680 x1818247334505600/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018695.809246:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334505600/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018695.809254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018695.809256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018695.809259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334505600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018695.809265:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334505728:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018695.809281:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018695.809287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018695/real 1734018695] req@ffff8800a78a0a80 x1818247334505728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018695.809298:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334505728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018695.809305:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018695.809307:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018695.809310:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334505728:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018695.809316:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334505856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018695.809328:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018695.809334:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018695/real 1734018695] req@ffff8800a78a1f80 x1818247334505856/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018695.809343:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334505856/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018695.809351:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018695.809353:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018695.809356:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334505856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018695.809362:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334505984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018695.809375:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018695.809381:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018695/real 1734018695] req@ffff8800a78a3b80 x1818247334505984/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018695.809397:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334505984/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018695.809404:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018695.809406:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018695.809409:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334505984:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018696.734769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018697.736744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018698.738763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018699.740799:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018700.742766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018700.816662:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018700.816675:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018700.816681:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018700.816695:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2967 00000100:00080000:2.0:1734018700.816707:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018700.816717:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018700.816768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334506112/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018700.816806:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018700.816811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018700.816813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018700.816818:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2967 00000100:00080000:2.0:1734018700.816821:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018700.816825:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018700.816836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018700.816849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334506240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018700.816861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018700.816864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018700.816866:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018700.816870:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2967 00000100:00080000:2.0:1734018700.816872:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018700.816876:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018700.816881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018700.816891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334506368/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018700.816900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018700.816903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018700.816904:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018700.816908:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2967 00000100:00080000:2.0:1734018700.816911:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018700.816914:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018700.816918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018700.816928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334506496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018700.816930:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334506112:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018700.816938:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018700.816941:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018700.816943:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018700.816946:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2967 00000100:00080000:2.0:1734018700.816950:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018700.816954:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018700.816958:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018700.816967:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334506624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018700.816975:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018700.816978:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018700.816980:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018700.816983:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2967 00000100:00080000:2.0:1734018700.816986:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018700.816990:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018700.816994:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018700.817002:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334506752/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018700.817003:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018700.817010:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018700.817013:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018700.817015:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018700.817018:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2967 00000100:00080000:2.0:1734018700.817021:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018700.817025:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018700.817026:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018700/real 1734018700] req@ffff8800a78a3b80 x1818247334506112/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018700.817029:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018700.817037:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334506880/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018700.817046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334506112/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018700.817058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018700.817062:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018700.817067:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334506112:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018700.817075:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334506240:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018700.817093:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018700.817100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018700/real 1734018700] req@ffff8800a78a1f80 x1818247334506240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018700.817112:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334506240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018700.817120:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018700.817124:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018700.817127:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334506240:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018700.817134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334506368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018700.817149:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018700.817155:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018700/real 1734018700] req@ffff8800a78a0a80 x1818247334506368/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018700.817165:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334506368/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018700.817173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018700.817176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018700.817180:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334506368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018700.817206:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334506496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018700.817219:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018700.817225:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018700/real 1734018700] req@ffff8800a78a2680 x1818247334506496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018700.817236:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334506496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018700.817244:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018700.817246:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018700.817249:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334506496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018700.817255:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334506624:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018700.817272:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018700.817278:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018700/real 1734018700] req@ffff8800a78a1880 x1818247334506624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018700.817289:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334506624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018700.817296:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018700.817299:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018700.817302:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334506624:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018700.817307:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334506752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018700.817320:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018700.817326:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018700/real 1734018700] req@ffff8800a78a0700 x1818247334506752/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018700.817336:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334506752/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018700.817343:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018700.817346:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018700.817349:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334506752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018700.817354:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334506880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018700.817367:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018700.817373:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018700/real 1734018700] req@ffff8800a78a3480 x1818247334506880/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018700.817389:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334506880/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018700.817396:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018700.817398:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018700.817401:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334506880:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018701.744798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018702.746769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018703.748767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018704.750767:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018705.752795:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018705.824617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018705.824630:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018705.824635:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018705.824649:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2972 00000100:00080000:2.0:1734018705.824660:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018705.824670:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018705.824721:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334507008/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018705.824759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018705.824763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018705.824765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018705.824770:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2972 00000100:00080000:2.0:1734018705.824774:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018705.824778:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018705.824789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018705.824801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334507136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018705.824811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018705.824814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018705.824816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018705.824821:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2972 00000100:00080000:2.0:1734018705.824823:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018705.824828:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018705.824834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018705.824844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334507264/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018705.824854:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018705.824857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018705.824859:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018705.824862:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2972 00000100:00080000:2.0:1734018705.824865:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018705.824868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018705.824873:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018705.824882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334507392/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018705.824892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018705.824896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018705.824897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018705.824899:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334507008:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018705.824901:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2972 00000100:00080000:2.0:1734018705.824904:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018705.824907:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018705.824911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018705.824920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334507520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018705.824928:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018705.824931:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018705.824933:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018705.824937:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2972 00000100:00080000:2.0:1734018705.824940:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018705.824944:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018705.824948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018705.824956:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334507648/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018705.824964:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018705.824967:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018705.824969:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018705.824972:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2972 00000100:00080000:2.0:1734018705.824975:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018705.824979:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018705.824983:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018705.824991:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334507776/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018705.825005:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334507136:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018705.825028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334507264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018705.825054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334507392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018705.825069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334507520:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018705.825087:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334507648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018705.825102:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334507776:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018706.754763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018707.756711:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018708.758667:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018708.830762:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734018708.830809:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018708.830829:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018708.830864:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018708.830873:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018708.830881:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018708.830888:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018708.830895:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018708.830901:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734018708.830969:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018705/real 1734018708] req@ffff8800a78a2680 x1818247334507392/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018708.831000:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334507392/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018708.831012:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018708.831016:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018708.831022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334507392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018708.831030:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018705/real 1734018708] req@ffff8800a78a0a80 x1818247334507520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018708.831046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334507520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018708.831059:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018708.831063:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018708.831069:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334507520:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018708.831079:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018705/real 1734018708] req@ffff8800a78a1f80 x1818247334507648/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018708.831096:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334507648/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018708.831109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018708.831114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018708.831119:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334507648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018708.831129:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018705/real 1734018708] req@ffff8800a78a3b80 x1818247334507776/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018708.831147:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334507776/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018708.831159:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018708.831164:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018708.831169:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334507776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018708.831183:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018705/real 1734018708] req@ffff8800a78a3480 x1818247334507008/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018708.831202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334507008/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018708.831214:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018708.831219:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018708.831225:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334507008:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018708.831235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018705/real 1734018708] req@ffff8800a78a0700 x1818247334507136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018708.831253:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334507136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018708.831266:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018708.831271:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018708.831276:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334507136:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018708.831287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018705/real 1734018708] req@ffff8800a78a1880 x1818247334507264/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018708.831306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334507264/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018708.831319:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018708.831333:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018708.831339:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334507264:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018709.760680:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018710.762769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018710.832606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018710.832618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018710.832624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018710.832638:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2977 00000100:00080000:2.0:1734018710.832649:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018710.832659:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018710.832710:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334507904/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018710.832759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018710.832763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018710.832765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018710.832770:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2977 00000100:00080000:2.0:1734018710.832773:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018710.832777:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018710.832788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018710.832801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334508032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018710.832810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018710.832813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018710.832815:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018710.832819:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2977 00000100:00080000:2.0:1734018710.832822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018710.832826:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018710.832831:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018710.832840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334508160/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018710.832849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018710.832853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018710.832854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018710.832858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2977 00000100:00080000:2.0:1734018710.832860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018710.832864:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018710.832869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018710.832877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334508288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018710.832887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018710.832890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018710.832892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018710.832896:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2977 00000100:00100000:1.0:1734018710.832897:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334507904:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018710.832898:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018710.832902:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018710.832906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018710.832915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334508416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018710.832923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018710.832926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018710.832928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018710.832932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2977 00000100:00080000:2.0:1734018710.832934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018710.832938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018710.832942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018710.832950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334508544/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018710.832958:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018710.832962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018710.832964:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018710.832968:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2977 00000100:00080000:2.0:1734018710.832971:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018710.832974:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018710.832975:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018710.832979:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018710.832987:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334508672/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018710.832997:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018710/real 1734018710] req@ffff8800a78a1880 x1818247334507904/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.833018:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334507904/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.833030:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.833034:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018710.833039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334507904:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018710.833047:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334508032:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018710.833066:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018710.833073:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018710/real 1734018710] req@ffff8800a78a0700 x1818247334508032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.833084:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334508032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.833092:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.833095:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018710.833099:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334508032:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018710.833105:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334508160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018710.833119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018710.833125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018710/real 1734018710] req@ffff8800a78a3480 x1818247334508160/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.833135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334508160/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.833144:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.833147:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018710.833150:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334508160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018710.833176:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334508288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018710.833190:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018710.833196:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018710/real 1734018710] req@ffff8800a78a3b80 x1818247334508288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.833206:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334508288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.833214:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.833217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018710.833220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334508288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018710.833226:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334508416:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018710.833238:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018710.833244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018710/real 1734018710] req@ffff8800a78a1f80 x1818247334508416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.833254:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334508416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.833262:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.833264:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018710.833267:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334508416:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018710.833274:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334508544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018710.833286:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018710.833292:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018710/real 1734018710] req@ffff8800a78a0a80 x1818247334508544/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.833302:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334508544/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.833309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.833312:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018710.833315:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334508544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018710.833320:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334508672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018710.833333:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018710.833339:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018710/real 1734018710] req@ffff8800a78a2680 x1818247334508672/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.833349:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334508672/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.833356:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.833358:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018710.833361:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334508672:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018711.764763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018712.766740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018713.768768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018714.770733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018715.772741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018715.840574:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018715.840584:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018715.840589:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018715.840599:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2982 00000100:00080000:2.0:1734018715.840608:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018715.840616:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018715.840654:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334508800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018715.840688:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018715.840693:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018715.840695:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018715.840699:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2982 00000100:00080000:2.0:1734018715.840703:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018715.840707:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018715.840716:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018715.840728:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334508928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018715.840740:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018715.840743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018715.840745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018715.840749:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2982 00000100:00080000:2.0:1734018715.840751:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018715.840755:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018715.840760:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018715.840768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334509056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018715.840777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018715.840780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018715.840782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018715.840785:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2982 00000100:00080000:2.0:1734018715.840788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018715.840791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018715.840795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018715.840804:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334508800:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018715.840804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334509184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018715.840814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018715.840816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018715.840818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018715.840821:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2982 00000100:00080000:2.0:1734018715.840825:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018715.840828:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018715.840833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018715.840840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334509312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018715.840848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018715.840851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018715.840853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018715.840856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2982 00000100:00080000:2.0:1734018715.840861:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018715.840864:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018715.840868:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018715.840872:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018715.840876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334509440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018715.840884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018715.840887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018715.840888:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018715.840891:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2982 00000100:00100000:1.0:1734018715.840893:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018715/real 1734018715] req@ffff8800a78a2680 x1818247334508800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018715.840894:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018715.840897:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018715.840901:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018715.840909:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334509568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018715.840914:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334508800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018715.840924:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018715.840927:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018715.840932:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334508800:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018715.840940:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334508928:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018715.840957:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018715.840964:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018715/real 1734018715] req@ffff8800a78a0a80 x1818247334508928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018715.840975:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334508928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018715.840982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018715.840985:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018715.840988:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334508928:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018715.840994:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334509056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018715.841007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018715.841013:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018715/real 1734018715] req@ffff8800a78a1f80 x1818247334509056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018715.841023:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334509056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018715.841030:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018715.841034:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018715.841037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334509056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018715.841062:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334509184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018715.841075:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018715.841081:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018715/real 1734018715] req@ffff8800a78a3b80 x1818247334509184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018715.841091:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334509184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018715.841099:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018715.841101:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018715.841104:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334509184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018715.841109:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334509312:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018715.841121:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018715.841127:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018715/real 1734018715] req@ffff8800a78a3480 x1818247334509312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018715.841137:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334509312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018715.841144:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018715.841146:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018715.841150:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334509312:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018715.841156:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334509440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018715.841181:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018715.841192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018715/real 1734018715] req@ffff8800a78a0700 x1818247334509440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018715.841208:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334509440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018715.841215:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018715.841217:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018715.841220:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334509440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018715.841225:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334509568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018715.841237:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018715.841243:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018715/real 1734018715] req@ffff8800a78a1880 x1818247334509568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018715.841252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334509568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018715.841259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018715.841262:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018715.841265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334509568:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018716.774736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018717.776792:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018718.778738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018719.780737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018720.782762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018720.848615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018720.848627:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018720.848633:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018720.848646:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2987 00000100:00080000:2.0:1734018720.848657:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018720.848667:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018720.848716:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334509696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018720.848755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018720.848759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018720.848761:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018720.848766:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2987 00000100:00080000:2.0:1734018720.848769:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018720.848774:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018720.848784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018720.848797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334509824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018720.848807:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018720.848810:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018720.848812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018720.848816:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2987 00000100:00080000:2.0:1734018720.848819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018720.848823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018720.848829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018720.848839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334509952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018720.848848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018720.848852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018720.848853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018720.848857:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2987 00000100:00080000:2.0:1734018720.848860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018720.848863:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018720.848867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018720.848876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334510080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018720.848885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018720.848888:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334509696:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018720.848888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018720.848890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018720.848893:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2987 00000100:00080000:2.0:1734018720.848896:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018720.848900:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018720.848904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018720.848913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334510208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018720.848921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018720.848924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018720.848926:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018720.848931:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2987 00000100:00080000:2.0:1734018720.848933:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018720.848937:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018720.848941:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018720.848949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334510336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018720.848957:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018720.848960:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018720.848962:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018720.848965:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018720.848966:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2987 00000100:00080000:2.0:1734018720.848969:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018720.848973:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018720.848977:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018720.848984:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334510464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018720.848989:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018720/real 1734018720] req@ffff8800a78a1880 x1818247334509696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018720.849017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334509696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018720.849028:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018720.849032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018720.849037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334509696:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018720.849045:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334509824:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018720.849063:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018720.849070:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018720/real 1734018720] req@ffff8800a78a0700 x1818247334509824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018720.849081:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334509824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018720.849089:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018720.849092:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018720.849095:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334509824:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018720.849102:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334509952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018720.849116:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018720.849123:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018720/real 1734018720] req@ffff8800a78a3480 x1818247334509952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018720.849133:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334509952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018720.849140:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018720.849144:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018720.849147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334509952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018720.849174:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334510080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018720.849187:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018720.849193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018720/real 1734018720] req@ffff8800a78a3b80 x1818247334510080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018720.849204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334510080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018720.849211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018720.849214:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018720.849217:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334510080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018720.849223:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334510208:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018720.849236:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018720.849242:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018720/real 1734018720] req@ffff8800a78a1f80 x1818247334510208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018720.849252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334510208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018720.849259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018720.849314:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018720.849319:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334510208:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018720.849326:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334510336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018720.849347:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018720.849354:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018720/real 1734018720] req@ffff8800a78a0a80 x1818247334510336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018720.849365:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334510336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018720.849373:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018720.849376:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018720.849379:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334510336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018720.849384:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334510464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018720.849398:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018720.849404:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018720/real 1734018720] req@ffff8800a78a2680 x1818247334510464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018720.849414:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334510464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018720.849421:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018720.849423:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018720.849426:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334510464:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018721.784720:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018722.786735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018723.788778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018724.790749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018725.792793:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018725.856619:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018725.856631:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018725.856637:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018725.856651:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2992 00000100:00080000:2.0:1734018725.856663:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018725.856672:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018725.856725:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334510592/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018725.856773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018725.856777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018725.856779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018725.856784:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2992 00000100:00080000:2.0:1734018725.856787:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018725.856791:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018725.856803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018725.856815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334510720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018725.856824:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018725.856827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018725.856829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018725.856833:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2992 00000100:00080000:2.0:1734018725.856836:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018725.856840:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018725.856845:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018725.856854:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334510848/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018725.856864:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018725.856867:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018725.856869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018725.856872:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2992 00000100:00080000:2.0:1734018725.856875:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018725.856879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018725.856884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018725.856892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334510976/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018725.856902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018725.856905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734018725.856906:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334510592:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018725.856907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018725.856910:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2992 00000100:00080000:2.0:1734018725.856913:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018725.856916:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018725.856920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018725.856929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334511104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018725.856937:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018725.856940:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018725.856941:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018725.856945:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2992 00000100:00080000:2.0:1734018725.856948:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018725.856951:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018725.856955:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018725.856964:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334511232/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018725.856973:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018725.856977:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018725.856978:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018725.856982:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018725.856982:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2992 00000100:00080000:2.0:1734018725.856986:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018725.856990:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018725.856994:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018725.857002:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334511360/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018725.857005:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018725/real 1734018725] req@ffff8800a78a2680 x1818247334510592/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018725.857034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334510592/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018725.857046:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018725.857050:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018725.857054:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334510592:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018725.857063:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334510720:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018725.857080:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018725.857087:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018725/real 1734018725] req@ffff8800a78a0a80 x1818247334510720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018725.857098:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334510720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018725.857106:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018725.857109:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018725.857112:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334510720:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018725.857119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334510848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018725.857133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018725.857139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018725/real 1734018725] req@ffff8800a78a1f80 x1818247334510848/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018725.857150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334510848/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018725.857157:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018725.857160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018725.857164:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334510848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018725.857223:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334510976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018725.857242:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018725.857249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018725/real 1734018725] req@ffff8800a78a3b80 x1818247334510976/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018725.857260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334510976/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018725.857268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018725.857271:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018725.857274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334510976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018725.857280:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334511104:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018725.857293:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018725.857299:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018725/real 1734018725] req@ffff8800a78a3480 x1818247334511104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018725.857309:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334511104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018725.857317:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018725.857319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018725.857322:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334511104:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018725.857328:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334511232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018725.857343:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018725.857350:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018725/real 1734018725] req@ffff8800a78a0700 x1818247334511232/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018725.857360:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334511232/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018725.857367:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018725.857370:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018725.857373:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334511232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018725.857378:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334511360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018725.857391:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018725.857398:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018725/real 1734018725] req@ffff8800a78a1880 x1818247334511360/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018725.857407:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334511360/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018725.857415:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018725.857417:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018725.857420:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334511360:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018726.794738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018727.796737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018728.798720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018729.800692:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018730.802757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018730.864609:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018730.864619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018730.864623:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018730.864634:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 2997 00000100:00080000:2.0:1734018730.864643:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018730.864651:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018730.864691:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334511488/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018730.864726:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018730.864730:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018730.864732:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018730.864737:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2997 00000100:00080000:2.0:1734018730.864740:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018730.864744:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018730.864754:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018730.864765:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334511616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018730.864777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018730.864780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018730.864782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018730.864786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2997 00000100:00080000:2.0:1734018730.864788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018730.864792:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018730.864796:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018730.864805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334511744/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018730.864814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018730.864817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018730.864818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018730.864822:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 2997 00000100:00080000:2.0:1734018730.864824:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018730.864827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018730.864832:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018730.864840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334511872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018730.864842:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334511488:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018730.864850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018730.864853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018730.864854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018730.864857:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2997 00000100:00080000:2.0:1734018730.864861:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018730.864864:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018730.864868:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018730.864876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334512000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018730.864883:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018730.864886:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018730.864888:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018730.864891:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2997 00000100:00080000:2.0:1734018730.864904:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018730.864908:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018730.864911:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734018730.864912:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018730.864920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334512128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018730.864928:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018730.864931:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018730.864932:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018730.864933:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018730/real 1734018730] req@ffff8800a78a1880 x1818247334511488/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018730.864936:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 2997 00000100:00080000:2.0:1734018730.864939:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018730.864942:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018730.864946:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018730.864954:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334512256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018730.864961:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334511488/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018730.864972:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018730.864975:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018730.864979:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334511488:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018730.864987:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334511616:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018730.865004:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018730.865011:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018730/real 1734018730] req@ffff8800a78a0700 x1818247334511616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018730.865021:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334511616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018730.865029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018730.865032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018730.865035:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334511616:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018730.865041:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334511744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018730.865054:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018730.865060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018730/real 1734018730] req@ffff8800a78a3480 x1818247334511744/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018730.865070:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334511744/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018730.865078:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018730.865081:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018730.865084:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334511744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018730.865141:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334511872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018730.865158:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018730.865166:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018730/real 1734018730] req@ffff8800a78a3b80 x1818247334511872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018730.865177:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334511872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018730.865184:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018730.865187:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018730.865190:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334511872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018730.865195:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334512000:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018730.865207:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018730.865213:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018730/real 1734018730] req@ffff8800a78a1f80 x1818247334512000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018730.865224:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334512000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018730.865231:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018730.865233:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018730.865236:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334512000:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018730.865241:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334512128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018730.865257:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018730.865263:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018730/real 1734018730] req@ffff8800a78a0a80 x1818247334512128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018730.865273:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334512128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018730.865280:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018730.865282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018730.865285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334512128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018730.865290:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334512256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018730.865303:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018730.865308:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018730/real 1734018730] req@ffff8800a78a2680 x1818247334512256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018730.865318:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334512256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018730.865325:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018730.865327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018730.865330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334512256:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018731.804739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018732.806752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018733.808719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018734.810760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018735.812742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018735.872614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018735.872623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018735.872627:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018735.872638:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3002 00000100:00080000:2.0:1734018735.872647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018735.872654:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018735.872695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334512384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018735.872730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018735.872734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018735.872736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018735.872740:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3002 00000100:00080000:2.0:1734018735.872743:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018735.872747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018735.872758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018735.872769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334512512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018735.872779:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018735.872782:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018735.872784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018735.872788:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3002 00000100:00080000:2.0:1734018735.872791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018735.872795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018735.872801:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018735.872810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334512640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018735.872819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018735.872822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018735.872824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018735.872827:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3002 00000100:00080000:2.0:1734018735.872830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018735.872832:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334512384:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018735.872833:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018735.872837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018735.872845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334512768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018735.872853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018735.872856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018735.872857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018735.872860:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3002 00000100:00080000:2.0:1734018735.872863:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018735.872866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018735.872870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018735.872877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334512896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018735.872886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018735.872888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018735.872891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018735.872895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3002 00000100:00080000:2.0:1734018735.872898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018735.872899:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018735.872901:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018735.872905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018735.872913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334513024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018735.872921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018735.872922:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018735/real 1734018735] req@ffff8800a78a2680 x1818247334512384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018735.872924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018735.872925:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018735.872928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3002 00000100:00080000:2.0:1734018735.872931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018735.872934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018735.872938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018735.872945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334513152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018735.872950:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334512384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018735.872960:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018735.872963:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018735.872968:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334512384:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018735.872975:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334512512:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018735.872991:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018735.872998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018735/real 1734018735] req@ffff8800a78a0a80 x1818247334512512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018735.873040:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334512512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018735.873048:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018735.873056:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018735.873059:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334512512:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018735.873084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334512640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018735.873101:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018735.873108:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018735/real 1734018735] req@ffff8800a78a1f80 x1818247334512640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018735.873118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334512640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018735.873126:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018735.873129:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018735.873132:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334512640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018735.873138:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334512768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018735.873151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018735.873157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018735/real 1734018735] req@ffff8800a78a3b80 x1818247334512768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018735.873167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334512768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018735.873175:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018735.873177:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018735.873180:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334512768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018735.873185:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334512896:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018735.873197:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018735.873203:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018735/real 1734018735] req@ffff8800a78a3480 x1818247334512896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018735.873213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334512896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018735.873220:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018735.873222:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018735.873226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334512896:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018735.873231:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334513024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018735.873247:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018735.873253:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018735/real 1734018735] req@ffff8800a78a0700 x1818247334513024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018735.873263:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334513024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018735.873270:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018735.873272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018735.873275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334513024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018735.873280:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334513152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018735.873292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018735.873298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018735/real 1734018735] req@ffff8800a78a1880 x1818247334513152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018735.873308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334513152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018735.873315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018735.873317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018735.873320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334513152:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018736.814710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018737.816719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018738.818724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018739.820776:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018740.822764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018740.880603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018740.880615:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018740.880621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018740.880635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3007 00000100:00080000:2.0:1734018740.880647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018740.880657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018740.880706:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334513280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018740.880752:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018740.880757:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018740.880758:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018740.880763:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3007 00000100:00080000:2.0:1734018740.880767:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018740.880771:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018740.880782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018740.880794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334513408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018740.880804:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018740.880807:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018740.880808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018740.880813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3007 00000100:00080000:2.0:1734018740.880816:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018740.880820:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018740.880824:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018740.880834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334513536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018740.880842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018740.880845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018740.880847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018740.880850:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3007 00000100:00080000:2.0:1734018740.880853:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018740.880857:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018740.880862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018740.880870:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334513280:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018740.880871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334513664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018740.880881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018740.880884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018740.880885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018740.880888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3007 00000100:00080000:2.0:1734018740.880891:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018740.880894:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018740.880898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018740.880907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334513792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018740.880915:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018740.880918:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018740.880954:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018740.880959:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3007 00000100:00080000:2.0:1734018740.880962:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018740.880966:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018740.880971:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018740.880980:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334513920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018740.880983:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018740.880989:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018740.880993:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018740.880995:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018740.880999:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3007 00000100:00080000:2.0:1734018740.881003:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018740.881007:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018740.881010:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018740/real 1734018740] req@ffff8800a78a1880 x1818247334513280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018740.881011:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018740.881019:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334514048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018740.881040:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334513280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018740.881052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018740.881056:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018740.881061:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334513280:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018740.881069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334513408:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018740.881088:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018740.881095:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018740/real 1734018740] req@ffff8800a78a0700 x1818247334513408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018740.881107:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334513408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018740.881114:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018740.881117:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018740.881120:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334513408:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018740.881126:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334513536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018740.881140:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018740.881146:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018740/real 1734018740] req@ffff8800a78a3480 x1818247334513536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018740.881156:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334513536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018740.881164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018740.881167:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018740.881170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334513536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018740.881199:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334513664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018740.881213:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018740.881219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018740/real 1734018740] req@ffff8800a78a3b80 x1818247334513664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018740.881230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334513664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018740.881238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018740.881240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018740.881243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334513664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018740.881248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334513792:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018740.881260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018740.881266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018740/real 1734018740] req@ffff8800a78a1f80 x1818247334513792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018740.881276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334513792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018740.881283:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018740.881285:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018740.881288:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334513792:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018740.881294:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334513920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018740.881310:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018740.881316:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018740/real 1734018740] req@ffff8800a78a0a80 x1818247334513920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018740.881325:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334513920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018740.881333:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018740.881335:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018740.881338:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334513920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018740.881343:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334514048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018740.881356:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018740.881362:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018740/real 1734018740] req@ffff8800a78a2680 x1818247334514048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018740.881371:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334514048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018740.881378:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018740.881380:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018740.881383:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334514048:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018741.824778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018742.826755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018743.828728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018744.830737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018745.832699:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018745.888607:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018745.888616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018745.888621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018745.888632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3012 00000100:00080000:2.0:1734018745.888641:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018745.888649:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018745.888686:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334514176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018745.888723:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018745.888727:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018745.888729:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018745.888734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3012 00000100:00080000:2.0:1734018745.888737:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018745.888741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018745.888751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018745.888762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334514304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018745.888774:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018745.888777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018745.888779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018745.888782:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3012 00000100:00080000:2.0:1734018745.888785:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018745.888788:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018745.888793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018745.888802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334514432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018745.888811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018745.888814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018745.888816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018745.888819:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3012 00000100:00080000:2.0:1734018745.888822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018745.888825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018745.888829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018745.888874:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334514560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018745.888884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018745.888887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018745.888889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018745.888893:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3012 00000100:00080000:2.0:1734018745.888896:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018745.888900:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018745.888904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018745.888914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334514688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018745.888915:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334514176:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018745.888923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018745.888927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018745.888928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018745.888932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3012 00000100:00080000:2.0:1734018745.888934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018745.888937:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018745.888942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018745.888949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334514816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018745.888957:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018745.888960:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018745.888962:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018745.888965:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3012 00000100:00080000:2.0:1734018745.888968:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018745.888971:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018745.888974:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018745.888982:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334514944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018745.889014:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018745.889043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018745/real 1734018745] req@ffff8800a78a2680 x1818247334514176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018745.889073:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334514176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018745.889085:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018745.889089:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018745.889095:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334514176:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018745.889103:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334514304:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018745.889121:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018745.889128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018745/real 1734018745] req@ffff8800a78a0a80 x1818247334514304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018745.889139:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334514304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018745.889147:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018745.889150:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018745.889153:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334514304:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018745.889160:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334514432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018745.889174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018745.889181:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018745/real 1734018745] req@ffff8800a78a1f80 x1818247334514432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018745.889191:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334514432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018745.889198:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018745.889202:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018745.889206:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334514432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018745.889213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334514560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018745.889227:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018745.889233:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018745/real 1734018745] req@ffff8800a78a3b80 x1818247334514560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018745.889243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334514560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018745.889250:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018745.889253:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018745.889256:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334514560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018745.889289:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334514688:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018745.889303:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018745.889310:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018745/real 1734018745] req@ffff8800a78a3480 x1818247334514688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018745.889320:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334514688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018745.889327:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018745.889329:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018745.889332:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334514688:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018745.889339:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334514816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018745.889355:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018745.889362:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018745/real 1734018745] req@ffff8800a78a0700 x1818247334514816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018745.889373:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334514816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018745.889380:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018745.889382:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018745.889385:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334514816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018745.889391:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334514944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018745.889404:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018745.889410:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018745/real 1734018745] req@ffff8800a78a1880 x1818247334514944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018745.889419:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334514944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018745.889426:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018745.889429:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018745.889432:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334514944:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018746.834683:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018747.836755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018748.838673:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018749.840744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018750.842799:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018750.896590:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018750.896599:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018750.896604:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018750.896615:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3017 00000100:00080000:2.0:1734018750.896624:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018750.896632:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018750.896672:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334515072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018750.896707:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018750.896711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018750.896713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018750.896718:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3017 00000100:00080000:2.0:1734018750.896721:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018750.896725:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018750.896735:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018750.896747:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334515200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018750.896783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018750.896786:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018750.896788:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018750.896793:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3017 00000100:00080000:2.0:1734018750.896796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018750.896800:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018750.896807:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018750.896817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334515328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018750.896826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018750.896829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018750.896831:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018750.896834:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3017 00000100:00080000:2.0:1734018750.896837:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018750.896840:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018750.896845:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018750.896853:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334515456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018750.896855:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334515072:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018750.896863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018750.896866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018750.896867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018750.896870:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3017 00000100:00080000:2.0:1734018750.896873:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018750.896876:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018750.896880:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018750.896888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334515584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018750.896896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018750.896899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018750.896901:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018750.896905:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3017 00000100:00080000:2.0:1734018750.896908:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018750.896911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018750.896915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018750.896923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334515712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018750.896931:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018750.896933:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018750.896935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018750.896938:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3017 00000100:00080000:2.0:1734018750.896941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018750.896944:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018750.896948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018750.896950:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018750.896956:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334515840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018750.896979:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018750/real 1734018750] req@ffff8800a78a1880 x1818247334515072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018750.897009:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334515072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018750.897021:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018750.897025:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018750.897030:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334515072:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018750.897038:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334515200:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018750.897056:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018750.897063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018750/real 1734018750] req@ffff8800a78a0700 x1818247334515200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018750.897074:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334515200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018750.897082:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018750.897085:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018750.897089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334515200:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018750.897095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334515328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018750.897110:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018750.897116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018750/real 1734018750] req@ffff8800a78a3480 x1818247334515328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018750.897126:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334515328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018750.897134:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018750.897137:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018750.897141:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334515328:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018750.897171:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334515456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018750.897186:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018750.897192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018750/real 1734018750] req@ffff8800a78a3b80 x1818247334515456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018750.897203:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334515456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018750.897210:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018750.897213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018750.897216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334515456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018750.897222:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334515584:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018750.897235:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018750.897241:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018750/real 1734018750] req@ffff8800a78a1f80 x1818247334515584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018750.897251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334515584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018750.897259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018750.897261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018750.897264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334515584:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018750.897270:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334515712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018750.897287:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018750.897293:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018750/real 1734018750] req@ffff8800a78a0a80 x1818247334515712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018750.897304:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334515712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018750.897311:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018750.897313:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018750.897316:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334515712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018750.897322:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334515840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018750.897335:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018750.897341:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018750/real 1734018750] req@ffff8800a78a2680 x1818247334515840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018750.897351:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334515840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018750.897358:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018750.897360:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018750.897363:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334515840:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018751.844784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018752.846784:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018753.848783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018754.850784:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018755.852753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018755.904587:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018755.904597:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018755.904601:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018755.904612:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3022 00000100:00080000:2.0:1734018755.904621:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018755.904629:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018755.904695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334515968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018755.904737:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018755.904741:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018755.904743:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018755.904748:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3022 00000100:00080000:2.0:1734018755.904752:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018755.904756:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018755.904766:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018755.904779:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334516096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018755.904789:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018755.904792:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018755.904793:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018755.904797:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3022 00000100:00080000:2.0:1734018755.904800:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018755.904803:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018755.904807:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018755.904816:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334516224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018755.904825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018755.904828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018755.904830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018755.904833:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3022 00000100:00080000:2.0:1734018755.904835:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018755.904839:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018755.904844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018755.904851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334516352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018755.904861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018755.904864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018755.904865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018755.904869:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3022 00000100:00100000:1.0:1734018755.904870:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334515968:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018755.904871:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018755.904874:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018755.904878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018755.904886:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334516480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018755.904894:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018755.904897:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018755.904898:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018755.904902:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3022 00000100:00080000:2.0:1734018755.904904:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018755.904907:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018755.904911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018755.904919:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334516608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018755.904927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018755.904930:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018755.904932:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018755.904935:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3022 00000100:00080000:2.0:1734018755.904938:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018755.904941:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018755.904945:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018755.904952:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334516736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018755.904966:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018755.904994:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018755/real 1734018755] req@ffff8800a78a2680 x1818247334515968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018755.905025:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334515968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018755.905037:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018755.905041:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018755.905046:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334515968:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018755.905054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334516096:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018755.905072:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018755.905080:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018755/real 1734018755] req@ffff8800a78a0a80 x1818247334516096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018755.905091:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334516096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018755.905099:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018755.905102:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018755.905105:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334516096:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018755.905112:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334516224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018755.905126:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018755.905133:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018755/real 1734018755] req@ffff8800a78a1f80 x1818247334516224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018755.905143:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334516224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018755.905150:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018755.905154:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018755.905157:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334516224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018755.905188:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334516352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018755.905202:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018755.905209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018755/real 1734018755] req@ffff8800a78a3b80 x1818247334516352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018755.905220:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334516352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018755.905228:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018755.905230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018755.905233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334516352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018755.905239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334516480:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018755.905253:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018755.905259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018755/real 1734018755] req@ffff8800a78a3480 x1818247334516480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018755.905269:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334516480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018755.905276:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018755.905278:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018755.905281:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334516480:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018755.905288:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334516608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018755.905304:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018755.905311:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018755/real 1734018755] req@ffff8800a78a0700 x1818247334516608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018755.905322:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334516608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018755.905329:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018755.905331:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018755.905334:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334516608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018755.905340:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334516736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018755.905353:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018755.905359:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018755/real 1734018755] req@ffff8800a78a1880 x1818247334516736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018755.905369:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334516736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018755.905376:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018755.905379:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018755.905382:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334516736:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018756.854744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018757.856718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018758.858769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018759.860809:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018760.862752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018760.912642:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018760.912654:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018760.912660:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018760.912673:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3027 00000100:00080000:2.0:1734018760.912684:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018760.912693:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018760.912744:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334516864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018760.912780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018760.912785:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018760.912787:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018760.912792:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3027 00000100:00080000:2.0:1734018760.912795:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018760.912799:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018760.912810:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018760.912823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334516992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018760.912834:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018760.912837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018760.912839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018760.912843:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3027 00000100:00080000:2.0:1734018760.912846:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018760.912849:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018760.912853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018760.912863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334517120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018760.912872:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018760.912875:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018760.912876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018760.912880:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3027 00000100:00080000:2.0:1734018760.912882:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018760.912886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018760.912890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018760.912900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334517248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018760.912902:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334516864:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018760.912909:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018760.912912:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018760.912914:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018760.912917:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3027 00000100:00080000:2.0:1734018760.912921:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018760.912924:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018760.912928:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018760.912937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334517376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018760.912945:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018760.912948:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018760.912949:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018760.912953:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3027 00000100:00080000:2.0:1734018760.912956:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018760.912959:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018760.912963:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018760.912971:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334517504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018760.912979:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018760.912983:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018760.912984:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018760.912988:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3027 00000800:00000100:1.0:1734018760.912989:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018760.912991:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018760.912994:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018760.912998:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018760.913006:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334517632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018760.913016:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018760/real 1734018760] req@ffff8800a78a1880 x1818247334516864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018760.913046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334516864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018760.913058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018760.913061:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018760.913066:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334516864:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018760.913074:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334516992:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018760.913092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018760.913099:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018760/real 1734018760] req@ffff8800a78a0700 x1818247334516992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018760.913110:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334516992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018760.913118:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018760.913120:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018760.913124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334516992:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018760.913130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334517120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018760.913144:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018760.913150:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018760/real 1734018760] req@ffff8800a78a3480 x1818247334517120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018760.913160:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334517120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018760.913167:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018760.913171:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018760.913174:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334517120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018760.913204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334517248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018760.913217:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018760.913224:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018760/real 1734018760] req@ffff8800a78a3b80 x1818247334517248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018760.913235:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334517248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018760.913242:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018760.913244:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018760.913247:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334517248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018760.913252:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334517376:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018760.913265:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018760.913270:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018760/real 1734018760] req@ffff8800a78a1f80 x1818247334517376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018760.913280:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334517376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018760.913287:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018760.913289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018760.913292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334517376:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018760.913298:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334517504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018760.913315:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018760.913321:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018760/real 1734018760] req@ffff8800a78a0a80 x1818247334517504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018760.913332:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334517504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018760.913339:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018760.913341:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018760.913344:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334517504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018760.913349:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334517632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018760.913362:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018760.913368:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018760/real 1734018760] req@ffff8800a78a2680 x1818247334517632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018760.913378:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334517632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018760.913384:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018760.913386:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018760.913389:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334517632:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018761.864817:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018762.866753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018763.868755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018764.870757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018765.872751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018765.920593:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018765.920603:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018765.920608:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018765.920618:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3032 00000100:00080000:2.0:1734018765.920627:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018765.920635:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018765.920675:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334517760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018765.920710:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018765.920714:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018765.920716:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018765.920721:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3032 00000100:00080000:2.0:1734018765.920724:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018765.920728:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018765.920738:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018765.920750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334517888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018765.920760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018765.920763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018765.920765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018765.920769:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3032 00000100:00080000:2.0:1734018765.920771:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018765.920775:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018765.920781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018765.920789:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334518016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018765.920798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018765.920802:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018765.920804:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018765.920807:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3032 00000100:00080000:2.0:1734018765.920809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018765.920812:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018765.920816:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018765.920824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334518144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018765.920832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018765.920835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018765.920836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018765.920840:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3032 00000100:00080000:2.0:1734018765.920842:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018765.920845:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018765.920849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018765.920857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334518272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018765.920867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018765.920870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1734018765.920872:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334517760:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018765.920872:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018765.920876:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3032 00000100:00080000:2.0:1734018765.920879:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018765.920882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018765.920886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018765.920895:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334518400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018765.920903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018765.920906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018765.920907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018765.920910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3032 00000100:00080000:2.0:1734018765.920913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018765.920916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018765.920920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018765.920927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334518528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018765.920972:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018765.921001:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018765/real 1734018765] req@ffff8800a78a2680 x1818247334517760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018765.921031:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334517760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018765.921043:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018765.921047:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018765.921052:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334517760:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018765.921061:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334517888:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018765.921079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018765.921086:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018765/real 1734018765] req@ffff8800a78a0a80 x1818247334517888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018765.921097:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334517888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018765.921105:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018765.921108:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018765.921112:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334517888:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018765.921118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334518016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018765.921133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018765.921139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018765/real 1734018765] req@ffff8800a78a1f80 x1818247334518016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018765.921149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334518016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018765.921157:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018765.921161:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018765.921164:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334518016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018765.921171:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334518144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018765.921185:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018765.921191:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018765/real 1734018765] req@ffff8800a78a3b80 x1818247334518144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018765.921201:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334518144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018765.921208:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018765.921211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018765.921214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334518144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018765.921246:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334518272:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018765.921260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018765.921266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018765/real 1734018765] req@ffff8800a78a3480 x1818247334518272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018765.921276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334518272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018765.921283:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018765.921286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018765.921289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334518272:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018765.921295:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334518400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018765.921312:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018765.921319:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018765/real 1734018765] req@ffff8800a78a0700 x1818247334518400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018765.921329:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334518400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018765.921336:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018765.921339:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018765.921342:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334518400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018765.921348:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334518528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018765.921360:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018765.921366:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018765/real 1734018765] req@ffff8800a78a1880 x1818247334518528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018765.921376:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334518528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018765.921383:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018765.921385:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018765.921388:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334518528:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018766.874756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018767.876686:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018768.878721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018769.880762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018770.882747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018770.928596:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018770.928606:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018770.928611:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018770.928621:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3037 00000100:00080000:2.0:1734018770.928630:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018770.928638:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018770.928677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334518656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018770.928722:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018770.928726:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018770.928727:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018770.928732:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3037 00000100:00080000:2.0:1734018770.928735:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018770.928739:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018770.928748:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018770.928760:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334518784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018770.928769:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018770.928772:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018770.928774:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018770.928778:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3037 00000100:00080000:2.0:1734018770.928781:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018770.928785:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018770.928789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018770.928798:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334518912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018770.928807:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018770.928810:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018770.928812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018770.928815:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3037 00000100:00080000:2.0:1734018770.928818:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018770.928822:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018770.928826:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018770.928835:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334519040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018770.928844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018770.928847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018770.928849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018770.928852:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3037 00000100:00080000:2.0:1734018770.928854:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018770.928858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018770.928862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018770.928866:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334518656:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018770.928870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334519168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018770.928878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018770.928881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018770.928882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018770.928885:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3037 00000100:00080000:2.0:1734018770.928888:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018770.928891:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018770.928895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018770.928903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334519296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018770.928911:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018770.928915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018770.928916:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018770.928920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3037 00000100:00080000:2.0:1734018770.928922:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018770.928925:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018770.928929:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018770.928936:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334519424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018770.928993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334518784:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018770.929017:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334518912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018770.929046:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334519040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018770.929064:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334519168:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018770.929082:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334519296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018770.929099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334519424:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018771.884744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018772.886754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018773.888716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018773.934748:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734018773.934795:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018773.934814:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018773.934850:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018773.934858:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018773.934866:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018773.934873:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018773.934879:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018773.934885:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734018773.934979:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018770/real 1734018773] req@ffff8800a78a3b80 x1818247334519040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018773.935030:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334519040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018773.935050:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018773.935057:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018773.935065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334519040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018773.935079:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018770/real 1734018773] req@ffff8800a78a1f80 x1818247334519168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018773.935100:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334519168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018773.935113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018773.935117:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018773.935123:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334519168:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018773.935135:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018770/real 1734018773] req@ffff8800a78a0a80 x1818247334519296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018773.935155:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334519296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018773.935169:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018773.935174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018773.935180:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334519296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018773.935192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018770/real 1734018773] req@ffff8800a78a2680 x1818247334519424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018773.935210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334519424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018773.935222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018773.935226:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018773.935232:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334519424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018773.935244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018770/real 1734018773] req@ffff8800a78a1880 x1818247334518656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018773.935264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334518656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018773.935276:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018773.935281:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018773.935286:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334518656:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018773.935297:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018770/real 1734018773] req@ffff8800a78a0700 x1818247334518784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018773.935316:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334518784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018773.935398:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018773.935403:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018773.935578:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334518784:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018773.935595:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018770/real 1734018773] req@ffff8800a78a3480 x1818247334518912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018773.935619:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334518912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018773.935633:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018773.935638:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018773.935644:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334518912:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018774.890754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018775.892736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018775.936611:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018775.936626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018775.936630:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018775.936641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3042 00000100:00080000:2.0:1734018775.936650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018775.936658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018775.936697:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334519552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018775.936733:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018775.936738:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018775.936740:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018775.936745:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3042 00000100:00080000:2.0:1734018775.936748:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018775.936752:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018775.936762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018775.936774:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334519680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018775.936785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018775.936788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018775.936789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018775.936793:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3042 00000100:00080000:2.0:1734018775.936796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018775.936799:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018775.936804:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018775.936813:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334519808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018775.936822:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018775.936826:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018775.936827:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018775.936831:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3042 00000100:00080000:2.0:1734018775.936833:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018775.936836:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018775.936840:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018775.936848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334519936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018775.936856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018775.936859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018775.936861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018775.936864:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3042 00000100:00080000:2.0:1734018775.936867:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018775.936871:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018775.936874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018775.936881:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334520064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018775.936891:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018775.936893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018775.936895:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018775.936899:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334519552:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018775.936899:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3042 00000100:00080000:2.0:1734018775.936901:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018775.936904:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018775.936908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018775.936916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334520192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018775.936924:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018775.936927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018775.936929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018775.936932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3042 00000100:00080000:2.0:1734018775.936934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018775.936937:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018775.936941:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018775.936949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334520320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018775.936999:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018775.937028:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018775/real 1734018775] req@ffff8800a78a3480 x1818247334519552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.937050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334519552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.937062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.937066:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018775.937071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334519552:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018775.937079:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334519680:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018775.937098:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018775.937106:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018775/real 1734018775] req@ffff8800a78a0700 x1818247334519680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.937118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334519680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.937126:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.937129:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018775.937132:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334519680:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018775.937139:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334519808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018775.937154:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018775.937160:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018775/real 1734018775] req@ffff8800a78a1880 x1818247334519808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.937171:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334519808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.937178:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.937181:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018775.937184:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334519808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018775.937190:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334519936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018775.937203:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018775.937209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018775/real 1734018775] req@ffff8800a78a2680 x1818247334519936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.937220:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334519936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.937235:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.937238:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018775.937241:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334519936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018775.937275:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334520064:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018775.937288:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018775.937295:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018775/real 1734018775] req@ffff8800a78a0a80 x1818247334520064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.937306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334520064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.937314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.937316:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018775.937319:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334520064:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018775.937371:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334520192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018775.937389:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018775.937396:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018775/real 1734018775] req@ffff8800a78a1f80 x1818247334520192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.937406:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334520192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.937415:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.937418:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018775.937421:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334520192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018775.937428:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334520320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018775.937442:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018775.937448:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018775/real 1734018775] req@ffff8800a78a3b80 x1818247334520320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.937459:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334520320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.937533:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.937536:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018775.937539:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334520320:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018776.894733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018777.896735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018778.898737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018779.900727:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018780.902668:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018780.944599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018780.944608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018780.944613:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018780.944624:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3047 00000100:00080000:2.0:1734018780.944632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018780.944640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018780.944679:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334520448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018780.944715:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018780.944719:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018780.944721:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018780.944725:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3047 00000100:00080000:2.0:1734018780.944728:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018780.944732:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018780.944742:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018780.944754:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334520576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018780.944764:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018780.944767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018780.944768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018780.944772:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3047 00000100:00080000:2.0:1734018780.944775:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018780.944779:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018780.944784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018780.944793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334520704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018780.944803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018780.944806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018780.944807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018780.944810:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3047 00000100:00080000:2.0:1734018780.944813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018780.944816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018780.944820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018780.944828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334520832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018780.944836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018780.944839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018780.944840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018780.944844:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3047 00000100:00080000:2.0:1734018780.944846:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018780.944849:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018780.944853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018780.944861:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334520960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018780.944868:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334520448:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018780.944871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018780.944874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018780.944877:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018780.944880:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3047 00000100:00080000:2.0:1734018780.944883:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018780.944886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018780.944890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018780.944898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334521088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018780.944905:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018780.944909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018780.944910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018780.944913:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3047 00000100:00080000:2.0:1734018780.944916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018780.944919:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018780.944923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018780.944930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334521216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018780.944969:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018780.944998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018780/real 1734018780] req@ffff8800a78a3b80 x1818247334520448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018780.945022:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334520448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018780.945035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018780.945038:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018780.945044:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334520448:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018780.945052:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334520576:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018780.945070:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018780.945078:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018780/real 1734018780] req@ffff8800a78a1f80 x1818247334520576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018780.945098:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334520576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018780.945106:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018780.945109:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018780.945112:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334520576:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018780.945119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334520704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018780.945134:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018780.945140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018780/real 1734018780] req@ffff8800a78a0a80 x1818247334520704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018780.945150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334520704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018780.945158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018780.945160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018780.945163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334520704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018780.945170:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334520832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018780.945184:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018780.945190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018780/real 1734018780] req@ffff8800a78a2680 x1818247334520832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018780.945200:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334520832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018780.945207:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018780.945211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018780.945214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334520832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018780.945279:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334520960:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018780.945298:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018780.945306:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018780/real 1734018780] req@ffff8800a78a1880 x1818247334520960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018780.945318:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334520960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018780.945326:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018780.945328:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018780.945332:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334520960:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018780.945338:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334521088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018780.945350:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018780.945356:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018780/real 1734018780] req@ffff8800a78a0700 x1818247334521088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018780.945366:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334521088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018780.945373:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018780.945376:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018780.945379:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334521088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018780.945385:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334521216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018780.945402:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018780.945409:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018780/real 1734018780] req@ffff8800a78a3480 x1818247334521216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018780.945419:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334521216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018780.945426:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018780.945429:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018780.945432:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334521216:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018781.904734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018782.906749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018783.908700:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018784.910738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018785.912768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018785.952583:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018785.952592:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018785.952596:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018785.952607:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3052 00000100:00080000:2.0:1734018785.952616:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018785.952623:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018785.952662:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334521344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018785.952705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018785.952709:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018785.952711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018785.952715:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3052 00000100:00080000:2.0:1734018785.952719:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018785.952722:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018785.952733:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018785.952744:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334521472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018785.952753:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018785.952756:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018785.952758:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018785.952762:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3052 00000100:00080000:2.0:1734018785.952764:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018785.952768:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018785.952773:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018785.952782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334521600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018785.952790:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018785.952793:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018785.952794:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018785.952797:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3052 00000100:00080000:2.0:1734018785.952800:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018785.952805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018785.952809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018785.952816:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334521728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018785.952825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018785.952828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018785.952829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018785.952832:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3052 00000100:00080000:2.0:1734018785.952835:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018785.952838:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018785.952842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018785.952849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334521856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018785.952858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018785.952860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018785.952862:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018785.952865:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3052 00000100:00080000:2.0:1734018785.952868:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018785.952871:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018785.952875:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334521344:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734018785.952875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018785.952883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334521984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018785.952891:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018785.952894:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018785.952896:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018785.952899:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3052 00000100:00080000:2.0:1734018785.952902:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018785.952905:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018785.952909:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018785.952917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334522112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018785.952974:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018785.953003:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018785/real 1734018785] req@ffff8800a78a3480 x1818247334521344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018785.953026:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334521344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018785.953040:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018785.953044:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018785.953049:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334521344:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018785.953057:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334521472:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018785.953076:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018785.953083:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018785/real 1734018785] req@ffff8800a78a0700 x1818247334521472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018785.953095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334521472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018785.953105:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018785.953108:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018785.953112:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334521472:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018785.953118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334521600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018785.953133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018785.953140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018785/real 1734018785] req@ffff8800a78a1880 x1818247334521600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018785.953150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334521600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018785.953200:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018785.953203:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018785.953207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334521600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018785.953214:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334521728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018785.953233:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018785.953240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018785/real 1734018785] req@ffff8800a78a2680 x1818247334521728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018785.953251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334521728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018785.953259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018785.953262:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018785.953265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334521728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018785.953299:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334521856:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018785.953314:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018785.953321:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018785/real 1734018785] req@ffff8800a78a0a80 x1818247334521856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018785.953332:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334521856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018785.953340:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018785.953342:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018785.953345:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334521856:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018785.953351:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334521984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018785.953365:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018785.953371:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018785/real 1734018785] req@ffff8800a78a1f80 x1818247334521984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018785.953380:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334521984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018785.953387:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018785.953390:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018785.953393:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334521984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018785.953399:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334522112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018785.953412:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018785.953419:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018785/real 1734018785] req@ffff8800a78a3b80 x1818247334522112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018785.953430:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334522112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018785.953438:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018785.953440:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018785.953443:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334522112:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018786.914756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018787.916836:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018788.918721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018789.920719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018790.922705:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018790.960612:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018790.960626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018790.960630:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018790.960642:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3057 00000100:00080000:2.0:1734018790.960651:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018790.960659:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018790.960699:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334522240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018790.960735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018790.960740:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018790.960742:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018790.960746:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3057 00000100:00080000:2.0:1734018790.960750:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018790.960753:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018790.960764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018790.960775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334522368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018790.960786:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018790.960789:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018790.960791:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018790.960795:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3057 00000100:00080000:2.0:1734018790.960797:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018790.960801:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018790.960805:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018790.960814:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334522496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018790.960823:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018790.960826:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018790.960827:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018790.960831:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3057 00000100:00080000:2.0:1734018790.960834:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018790.960837:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018790.960841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018790.960849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334522624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018790.960857:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018790.960860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018790.960862:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018790.960865:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3057 00000100:00080000:2.0:1734018790.960868:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018790.960871:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018790.960875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018790.960883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334522752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018790.960892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018790.960895:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018790.960896:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018790.960900:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3057 00000100:00080000:2.0:1734018790.960902:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018790.960905:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018790.960908:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334522240:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734018790.960910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018790.960917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334522880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018790.960925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018790.960928:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018790.960929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018790.960933:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3057 00000100:00080000:2.0:1734018790.960935:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018790.960938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018790.960943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018790.960950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334523008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018790.961008:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018790.961037:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018790/real 1734018790] req@ffff8800a78a3b80 x1818247334522240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018790.961059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334522240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018790.961118:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018790.961122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018790.961128:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334522240:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018790.961137:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334522368:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018790.961161:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018790.961169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018790/real 1734018790] req@ffff8800a78a1f80 x1818247334522368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018790.961181:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334522368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018790.961189:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018790.961194:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018790.961197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334522368:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018790.961204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334522496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018790.961220:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018790.961226:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018790/real 1734018790] req@ffff8800a78a0a80 x1818247334522496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018790.961236:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334522496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018790.961245:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018790.961247:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018790.961251:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334522496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018790.961258:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334522624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018790.961271:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018790.961277:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018790/real 1734018790] req@ffff8800a78a2680 x1818247334522624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018790.961287:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334522624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018790.961294:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018790.961297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018790.961300:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334522624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018790.961332:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334522752:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018790.961350:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018790.961356:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018790/real 1734018790] req@ffff8800a78a1880 x1818247334522752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018790.961368:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334522752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018790.961376:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018790.961378:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018790.961381:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334522752:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018790.961387:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334522880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018790.961400:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018790.961406:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018790/real 1734018790] req@ffff8800a78a0700 x1818247334522880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018790.961416:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334522880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018790.961423:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018790.961425:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018790.961428:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334522880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018790.961434:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334523008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018790.961447:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018790.961453:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018790/real 1734018790] req@ffff8800a78a3480 x1818247334523008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018790.961580:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334523008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018790.961588:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018790.961591:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018790.961594:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334523008:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018791.924779:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018792.926748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018793.928754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018794.930794:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018795.932671:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018795.968544:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018795.968551:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018795.968554:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018795.968562:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3062 00000100:00080000:2.0:1734018795.968569:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018795.968574:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018795.968602:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334523136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018795.968623:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018795.968626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018795.968627:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018795.968630:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3062 00000100:00080000:2.0:1734018795.968632:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018795.968634:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018795.968641:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018795.968648:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334523264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018795.968654:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018795.968656:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018795.968657:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018795.968659:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3062 00000100:00080000:2.0:1734018795.968661:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018795.968664:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018795.968668:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018795.968674:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334523392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018795.968679:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018795.968681:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018795.968682:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018795.968684:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3062 00000100:00080000:2.0:1734018795.968685:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018795.968687:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018795.968690:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018795.968694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334523520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018795.968699:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018795.968700:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334523136:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018795.968701:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018795.968702:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018795.968704:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3062 00000100:00080000:2.0:1734018795.968706:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018795.968707:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018795.968710:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018795.968714:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334523648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018795.968719:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018795.968721:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018795.968723:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018795.968725:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3062 00000100:00080000:2.0:1734018795.968727:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018795.968729:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018795.968731:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018795.968736:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334523776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018795.968740:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018795.968742:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018795.968743:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018795.968745:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3062 00000100:00080000:2.0:1734018795.968746:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018795.968748:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018795.968748:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018795.968751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018795.968755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334523904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018795.968762:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018795/real 1734018795] req@ffff8800a78a3480 x1818247334523136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018795.968774:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334523136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018795.968780:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018795.968782:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018795.968785:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334523136:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018795.968789:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334523264:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018795.968799:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018795.968803:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018795/real 1734018795] req@ffff8800a78a0700 x1818247334523264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018795.968809:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334523264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018795.968814:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018795.968816:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018795.968818:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334523264:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018795.968822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334523392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018795.968831:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018795.968834:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018795/real 1734018795] req@ffff8800a78a1880 x1818247334523392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018795.968840:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334523392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018795.968844:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018795.968846:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018795.968848:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334523392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018795.968864:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334523520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018795.968871:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018795.968875:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018795/real 1734018795] req@ffff8800a78a2680 x1818247334523520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018795.968881:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334523520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018795.968886:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018795.968887:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018795.968889:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334523520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018795.968892:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334523648:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018795.968901:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018795.968905:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018795/real 1734018795] req@ffff8800a78a0a80 x1818247334523648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018795.968911:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334523648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018795.968915:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018795.968916:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018795.968918:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334523648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018795.968921:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334523776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018795.968928:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018795.968931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018795/real 1734018795] req@ffff8800a78a1f80 x1818247334523776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018795.968937:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334523776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018795.968942:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018795.968943:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018795.968945:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334523776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018795.968948:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334523904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018795.968955:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018795.968959:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018795/real 1734018795] req@ffff8800a78a3b80 x1818247334523904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018795.968968:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334523904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018795.968972:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018795.968974:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018795.968975:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334523904:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018796.934741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018797.936760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018798.938729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018799.940768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018800.942717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018800.976615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018800.976624:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018800.976629:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018800.976640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3067 00000100:00080000:2.0:1734018800.976649:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018800.976657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018800.976697:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334524032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018800.976739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018800.976743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018800.976745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018800.976749:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3067 00000100:00080000:2.0:1734018800.976752:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018800.976756:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018800.976766:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018800.976778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334524160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018800.976787:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018800.976790:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018800.976792:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018800.976796:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3067 00000100:00080000:2.0:1734018800.976799:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018800.976803:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018800.976807:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018800.976816:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334524288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018800.976825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018800.976828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018800.976829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018800.976832:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3067 00000100:00080000:2.0:1734018800.976835:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018800.976839:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018800.976843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018800.976847:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334524032:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018800.976852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334524416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018800.976861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018800.976864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018800.976865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018800.976868:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3067 00000100:00080000:2.0:1734018800.976871:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018800.976874:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018800.976878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018800.976885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334524544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018800.976893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018800.976896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018800.976897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018800.976943:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3067 00000800:00000100:1.0:1734018800.976945:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018800.976947:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018800.976950:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018800.976954:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018800.976963:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334524672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018800.976968:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018800/real 1734018800] req@ffff8800a78a3b80 x1818247334524032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018800.976972:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018800.976976:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018800.976978:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018800.976981:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3067 00000100:00080000:2.0:1734018800.976984:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018800.976987:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018800.976988:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334524032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018800.976991:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018800.976998:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1734018800.976998:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334524800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018800.977002:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018800.977006:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334524032:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018800.977014:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334524160:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018800.977033:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018800.977040:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018800/real 1734018800] req@ffff8800a78a1f80 x1818247334524160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018800.977051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334524160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018800.977059:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018800.977063:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018800.977066:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334524160:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018800.977073:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334524288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018800.977086:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018800.977092:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018800/real 1734018800] req@ffff8800a78a0a80 x1818247334524288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018800.977102:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334524288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018800.977110:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018800.977112:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018800.977116:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334524288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018800.977140:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334524416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018800.977153:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018800.977159:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018800/real 1734018800] req@ffff8800a78a2680 x1818247334524416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018800.977169:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334524416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018800.977177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018800.977179:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018800.977182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334524416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018800.977187:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334524544:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018800.977203:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018800.977209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018800/real 1734018800] req@ffff8800a78a1880 x1818247334524544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018800.977220:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334524544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018800.977227:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018800.977229:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018800.977232:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334524544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018800.977237:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334524672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018800.977248:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018800.977254:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018800/real 1734018800] req@ffff8800a78a0700 x1818247334524672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018800.977264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334524672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018800.977271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018800.977273:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018800.977276:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334524672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018800.977281:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334524800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018800.977293:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018800.977299:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018800/real 1734018800] req@ffff8800a78a3480 x1818247334524800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018800.977316:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334524800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018800.977324:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018800.977326:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018800.977329:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334524800:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018801.944753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018802.946694:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018803.948735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018804.950725:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018805.952748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018805.984593:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018805.984602:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018805.984607:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018805.984618:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3072 00000100:00080000:2.0:1734018805.984627:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018805.984634:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018805.984674:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334524928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018805.984708:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018805.984712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018805.984714:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018805.984718:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3072 00000100:00080000:2.0:1734018805.984722:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018805.984725:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018805.984736:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018805.984748:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334525056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018805.984758:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018805.984761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018805.984763:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018805.984767:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3072 00000100:00080000:2.0:1734018805.984770:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018805.984773:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018805.984778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018805.984787:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334525184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018805.984795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018805.984798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018805.984800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018805.984803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3072 00000100:00080000:2.0:1734018805.984805:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018805.984809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018805.984812:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334524928:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734018805.984841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018805.984851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334525312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018805.984861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018805.984864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018805.984866:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018805.984869:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3072 00000100:00080000:2.0:1734018805.984873:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018805.984876:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018805.984881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018805.984888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334525440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018805.984896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018805.984899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018805.984901:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018805.984904:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3072 00000100:00080000:2.0:1734018805.984907:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018805.984910:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018805.984914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734018805.984914:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018805.984922:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334525568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018805.984929:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018805.984932:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018805.984934:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018805.984935:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018805/real 1734018805] req@ffff8800a78a3480 x1818247334524928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018805.984937:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3072 00000100:00080000:2.0:1734018805.984940:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018805.984943:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018805.984947:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018805.984954:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334525696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018805.984956:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334524928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018805.984966:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018805.984970:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018805.984974:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334524928:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018805.984982:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334525056:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018805.984998:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018805.985004:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018805/real 1734018805] req@ffff8800a78a0700 x1818247334525056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018805.985015:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334525056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018805.985023:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018805.985027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018805.985031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334525056:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018805.985037:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334525184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018805.985050:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018805.985056:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018805/real 1734018805] req@ffff8800a78a1880 x1818247334525184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018805.985066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334525184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018805.985074:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018805.985077:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018805.985081:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334525184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018805.985105:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334525312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018805.985117:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018805.985123:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018805/real 1734018805] req@ffff8800a78a2680 x1818247334525312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018805.985134:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334525312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018805.985142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018805.985144:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018805.985147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334525312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018805.985152:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334525440:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018805.985168:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018805.985174:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018805/real 1734018805] req@ffff8800a78a0a80 x1818247334525440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018805.985184:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334525440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018805.985191:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018805.985193:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018805.985196:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334525440:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018805.985201:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334525568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018805.985212:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018805.985218:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018805/real 1734018805] req@ffff8800a78a1f80 x1818247334525568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018805.985228:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334525568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018805.985235:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018805.985237:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018805.985240:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334525568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018805.985245:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334525696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018805.985258:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018805.985264:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018805/real 1734018805] req@ffff8800a78a3b80 x1818247334525696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018805.985280:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334525696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018805.985288:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018805.985290:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018805.985293:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334525696:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018806.954715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018807.956757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018808.958807:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018809.960785:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018810.962787:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018810.992602:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018810.992611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018810.992616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018810.992626:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3077 00000100:00080000:2.0:1734018810.992635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018810.992643:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018810.992681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334525824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018810.992716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018810.992720:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018810.992722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018810.992726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3077 00000100:00080000:2.0:1734018810.992754:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018810.992759:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018810.992769:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018810.992781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334525952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018810.992792:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018810.992795:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018810.992797:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018810.992801:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3077 00000100:00080000:2.0:1734018810.992803:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018810.992807:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018810.992813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018810.992822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334526080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018810.992827:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334525824:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018810.992832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018810.992835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018810.992837:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018810.992840:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3077 00000100:00080000:2.0:1734018810.992843:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018810.992846:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018810.992850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018810.992858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334526208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018810.992866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018810.992869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018810.992870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018810.992874:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3077 00000100:00080000:2.0:1734018810.992876:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018810.992879:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018810.992883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018810.992891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334526336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018810.992894:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018810.992898:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018810.992901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018810.992903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018810.992907:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3077 00000100:00080000:2.0:1734018810.992911:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018810.992914:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018810.992915:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018810/real 1734018810] req@ffff8800a78a3b80 x1818247334525824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018810.992918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018810.992925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334526464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018810.992933:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018810.992934:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334525824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018810.992936:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018810.992938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018810.992941:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3077 00000100:00080000:2.0:1734018810.992943:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734018810.992945:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018810.992946:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734018810.992948:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000008:00100000:2.0:1734018810.992950:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018810.992953:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334525824:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018810.992957:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334526592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018810.992961:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334525952:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018810.992977:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018810.992983:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018810/real 1734018810] req@ffff8800a78a1f80 x1818247334525952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018810.992995:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334525952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018810.993003:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018810.993006:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018810.993010:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334525952:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018810.993033:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334526080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018810.993047:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018810.993054:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018810/real 1734018810] req@ffff8800a78a0a80 x1818247334526080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018810.993064:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334526080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018810.993072:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018810.993074:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018810.993077:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334526080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018810.993082:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334526208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018810.993094:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018810.993100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018810/real 1734018810] req@ffff8800a78a2680 x1818247334526208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018810.993110:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334526208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018810.993117:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018810.993119:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018810.993122:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334526208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018810.993128:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334526336:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018810.993143:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018810.993149:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018810/real 1734018810] req@ffff8800a78a1880 x1818247334526336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018810.993159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334526336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018810.993166:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018810.993168:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018810.993171:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334526336:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018810.993176:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334526464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018810.993188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018810.993194:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018810/real 1734018810] req@ffff8800a78a0700 x1818247334526464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018810.993203:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334526464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018810.993210:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018810.993213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018810.993216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334526464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018810.993221:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334526592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018810.993232:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018810.993238:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018810/real 1734018810] req@ffff8800a78a3480 x1818247334526592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018810.993255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334526592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018810.993262:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018810.993264:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018810.993267:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334526592:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018811.964692:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018812.966770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018813.968753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018814.970747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018815.972810:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018816.000612:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018816.000622:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018816.000626:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018816.000637:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3082 00000100:00080000:2.0:1734018816.000671:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018816.000680:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018816.000718:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334526720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018816.000760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018816.000764:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018816.000765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018816.000770:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3082 00000100:00080000:2.0:1734018816.000773:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018816.000777:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018816.000787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018816.000799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334526848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018816.000808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018816.000811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018816.000813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018816.000817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3082 00000100:00080000:2.0:1734018816.000819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018816.000823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018816.000828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018816.000836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334526976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018816.000845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018816.000848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018816.000850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018816.000853:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3082 00000100:00080000:2.0:1734018816.000855:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018816.000859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018816.000863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018816.000871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334527104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018816.000873:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334526720:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018816.000880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018816.000883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018816.000885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018816.000888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3082 00000100:00080000:2.0:1734018816.000890:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018816.000894:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018816.000898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018816.000905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334527232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018816.000913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018816.000916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018816.000917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018816.000921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3082 00000100:00080000:2.0:1734018816.000923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018816.000926:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018816.000930:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018816.000938:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334527360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018816.000939:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018816.000946:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018816.000950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018816.000951:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018816.000955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3082 00000100:00080000:2.0:1734018816.000958:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018816.000961:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018816/real 1734018816] req@ffff8800a78a3480 x1818247334526720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018816.000962:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018816.000966:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018816.000973:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334527488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018816.000981:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334526720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018816.000992:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018816.000995:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018816.001000:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334526720:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018816.001007:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334526848:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018816.001024:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018816.001031:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018816/real 1734018816] req@ffff8800a78a0700 x1818247334526848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018816.001042:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334526848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018816.001050:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018816.001053:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018816.001057:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334526848:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018816.001063:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334526976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018816.001076:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018816.001082:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018816/real 1734018816] req@ffff8800a78a1880 x1818247334526976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018816.001093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334526976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018816.001100:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018816.001103:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018816.001106:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334526976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018816.001129:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334527104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018816.001142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018816.001148:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018816/real 1734018816] req@ffff8800a78a2680 x1818247334527104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018816.001159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334527104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018816.001166:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018816.001168:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018816.001171:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334527104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018816.001177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334527232:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018816.001192:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018816.001199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018816/real 1734018816] req@ffff8800a78a0a80 x1818247334527232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018816.001209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334527232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018816.001216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018816.001218:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018816.001221:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334527232:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018816.001226:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334527360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018816.001238:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018816.001244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018816/real 1734018816] req@ffff8800a78a1f80 x1818247334527360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018816.001253:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334527360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018816.001260:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018816.001262:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018816.001265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334527360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018816.001270:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334527488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018816.001282:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018816.001288:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018816/real 1734018816] req@ffff8800a78a3b80 x1818247334527488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018816.001305:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334527488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018816.001313:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018816.001315:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018816.001318:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334527488:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018816.974747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018817.976784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018818.978802:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018819.980819:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018820.982776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018821.008617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018821.008630:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018821.008636:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018821.008650:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3087 00000100:00080000:2.0:1734018821.008660:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018821.008669:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018821.008719:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334527616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018821.008755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018821.008759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018821.008761:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018821.008766:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3087 00000100:00080000:2.0:1734018821.008769:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018821.008773:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018821.008784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018821.008797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334527744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018821.008808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018821.008812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018821.008814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018821.008818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3087 00000100:00080000:2.0:1734018821.008820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018821.008824:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018821.008828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018821.008837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334527872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018821.008847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018821.008850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018821.008852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018821.008855:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3087 00000100:00080000:2.0:1734018821.008858:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018821.008861:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018821.008866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018821.008874:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334528000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018821.008884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018821.008887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734018821.008889:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334527616:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018821.008889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018821.008893:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3087 00000100:00080000:2.0:1734018821.008897:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018821.008901:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018821.008905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018821.008914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334528128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018821.008922:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018821.008925:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018821.008927:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018821.008930:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3087 00000100:00080000:2.0:1734018821.008933:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018821.008936:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018821.008940:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018821.008948:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334528256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018821.008956:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018821.008959:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018821.008961:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018821.008963:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018821.008964:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3087 00000100:00080000:2.0:1734018821.008968:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018821.008971:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018821.008975:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018821.008983:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334528384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018821.008986:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018821/real 1734018821] req@ffff8800a78a3b80 x1818247334527616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018821.009007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334527616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018821.009019:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018821.009023:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018821.009028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334527616:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018821.009036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334527744:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018821.009054:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018821.009061:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018821/real 1734018821] req@ffff8800a78a1f80 x1818247334527744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018821.009073:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334527744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018821.009081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018821.009085:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018821.009088:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334527744:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018821.009095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334527872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018821.009109:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018821.009115:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018821/real 1734018821] req@ffff8800a78a0a80 x1818247334527872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018821.009126:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334527872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018821.009134:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018821.009136:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018821.009140:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334527872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018821.009168:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334528000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018821.009181:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018821.009188:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018821/real 1734018821] req@ffff8800a78a2680 x1818247334528000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018821.009198:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334528000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018821.009206:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018821.009208:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018821.009211:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334528000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018821.009217:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334528128:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018821.009234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018821.009240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018821/real 1734018821] req@ffff8800a78a1880 x1818247334528128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018821.009251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334528128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018821.009258:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018821.009261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018821.009264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334528128:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018821.009269:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334528256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018821.009281:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018821.009287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018821/real 1734018821] req@ffff8800a78a0700 x1818247334528256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018821.009298:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334528256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018821.009305:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018821.009308:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018821.009311:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334528256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018821.009317:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334528384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018821.009330:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018821.009336:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018821/real 1734018821] req@ffff8800a78a3480 x1818247334528384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018821.009352:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334528384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018821.009359:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018821.009361:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018821.009364:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334528384:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018821.984807:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018822.986751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018823.988749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018824.990777:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018825.992715:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018826.016575:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018826.016585:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018826.016590:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018826.016602:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3092 00000100:00080000:2.0:1734018826.016610:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018826.016618:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018826.016656:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334528512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018826.016690:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018826.016694:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018826.016696:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018826.016701:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3092 00000100:00080000:2.0:1734018826.016704:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018826.016708:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018826.016718:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018826.016730:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334528640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018826.016740:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018826.016743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018826.016744:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018826.016748:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3092 00000100:00080000:2.0:1734018826.016751:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018826.016755:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018826.016761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018826.016770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334528768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018826.016779:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018826.016782:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018826.016783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018826.016786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3092 00000100:00080000:2.0:1734018826.016789:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018826.016792:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018826.016796:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018826.016799:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334528512:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018826.016804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334528896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018826.016813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018826.016816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018826.016817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018826.016821:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3092 00000100:00080000:2.0:1734018826.016823:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018826.016826:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018826.016830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018826.016837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334529024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018826.016845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018826.016848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018826.016851:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018826.016854:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3092 00000100:00080000:2.0:1734018826.016857:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018826.016860:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018826.016864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018826.016866:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018826.016872:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334529152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018826.016880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018826.016883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018826.016884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018826.016888:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018826/real 1734018826] req@ffff8800a78a3480 x1818247334528512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018826.016888:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3092 00000100:00080000:2.0:1734018826.016890:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018826.016893:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018826.016897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018826.016905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334529280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018826.016907:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334528512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018826.016919:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018826.016922:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018826.016926:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334528512:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018826.016934:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334528640:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018826.016950:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018826.016957:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018826/real 1734018826] req@ffff8800a78a0700 x1818247334528640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018826.016969:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334528640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018826.016976:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018826.016980:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018826.016984:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334528640:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018826.016990:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334528768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018826.017003:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018826.017009:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018826/real 1734018826] req@ffff8800a78a1880 x1818247334528768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018826.017019:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334528768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018826.017027:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018826.017029:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018826.017033:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334528768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018826.017058:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334528896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018826.017071:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018826.017077:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018826/real 1734018826] req@ffff8800a78a2680 x1818247334528896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018826.017087:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334528896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018826.017095:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018826.017097:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018826.017100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334528896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018826.017106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334529024:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018826.017121:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018826.017128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018826/real 1734018826] req@ffff8800a78a0a80 x1818247334529024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018826.017138:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334529024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018826.017145:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018826.017147:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018826.017150:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334529024:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018826.017155:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334529152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018826.017167:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018826.017172:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018826/real 1734018826] req@ffff8800a78a1f80 x1818247334529152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018826.017182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334529152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018826.017189:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018826.017191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018826.017194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334529152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018826.017199:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334529280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018826.017211:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018826.017218:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018826/real 1734018826] req@ffff8800a78a3b80 x1818247334529280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018826.017234:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334529280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018826.017242:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018826.017244:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018826.017247:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334529280:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018826.994736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018827.996744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018828.998694:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018830.000779:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018831.002705:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018831.024610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018831.024619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018831.024624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018831.024634:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3097 00000100:00080000:2.0:1734018831.024643:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018831.024651:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018831.024689:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334529408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018831.024731:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018831.024735:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018831.024737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018831.024741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3097 00000100:00080000:2.0:1734018831.024744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018831.024748:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018831.024759:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018831.024771:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334529536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018831.024781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018831.024784:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018831.024785:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018831.024789:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3097 00000100:00080000:2.0:1734018831.024792:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018831.024796:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018831.024800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018831.024809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334529664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018831.024817:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018831.024820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018831.024822:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018831.024825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3097 00000100:00080000:2.0:1734018831.024827:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018831.024831:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018831.024835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018831.024843:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334529408:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018831.024844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334529792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018831.024852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018831.024855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018831.024857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018831.024860:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3097 00000100:00080000:2.0:1734018831.024862:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018831.024865:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018831.024869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018831.024877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334529920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018831.024885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018831.024887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018831.024889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018831.024892:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3097 00000100:00080000:2.0:1734018831.024904:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018831.024908:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018831.024908:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018831.024912:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018831.024920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334530048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018831.024928:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018831.024930:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018831/real 1734018831] req@ffff8800a78a3b80 x1818247334529408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018831.024931:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018831.024933:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018831.024937:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3097 00000100:00080000:2.0:1734018831.024940:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018831.024943:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018831.024947:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018831.024950:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334529408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018831.024955:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334530176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018831.024960:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018831.024964:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018831.024969:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334529408:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018831.024976:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334529536:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018831.024993:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018831.025000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018831/real 1734018831] req@ffff8800a78a1f80 x1818247334529536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018831.025011:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334529536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018831.025019:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018831.025022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018831.025026:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334529536:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018831.025032:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334529664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018831.025046:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018831.025052:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018831/real 1734018831] req@ffff8800a78a0a80 x1818247334529664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018831.025062:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334529664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018831.025069:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018831.025072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018831.025075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334529664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018831.025099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334529792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018831.025112:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018831.025118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018831/real 1734018831] req@ffff8800a78a2680 x1818247334529792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018831.025128:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334529792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018831.025136:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018831.025138:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018831.025141:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334529792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018831.025147:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334529920:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018831.025163:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018831.025169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018831/real 1734018831] req@ffff8800a78a1880 x1818247334529920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018831.025179:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334529920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018831.025186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018831.025188:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018831.025191:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334529920:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018831.025196:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334530048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018831.025207:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018831.025213:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018831/real 1734018831] req@ffff8800a78a0700 x1818247334530048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018831.025223:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334530048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018831.025230:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018831.025232:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018831.025236:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334530048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018831.025241:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334530176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018831.025254:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018831.025260:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018831/real 1734018831] req@ffff8800a78a3480 x1818247334530176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018831.025276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334530176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018831.025283:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018831.025285:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018831.025288:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334530176:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018832.004764:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018833.006742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018834.008741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018835.010719:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018836.012727:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018836.032586:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018836.032595:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018836.032600:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018836.032610:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3102 00000100:00080000:2.0:1734018836.032619:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018836.032662:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018836.032700:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334530304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018836.032734:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018836.032739:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018836.032741:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018836.032745:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3102 00000100:00080000:2.0:1734018836.032749:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018836.032753:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018836.032762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018836.032790:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334530432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018836.032811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018836.032814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018836.032816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018836.032820:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3102 00000100:00080000:2.0:1734018836.032823:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018836.032827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018836.032832:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018836.032841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334530560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018836.032851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018836.032852:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334530304:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018836.032854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018836.032856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018836.032859:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3102 00000100:00080000:2.0:1734018836.032862:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018836.032865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018836.032869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018836.032877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334530688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018836.032885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018836.032888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018836.032890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018836.032893:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3102 00000100:00080000:2.0:1734018836.032897:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018836.032900:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018836.032904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018836.032911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334530816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018836.032919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018836.032922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018836.032924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018836.032927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3102 00000100:00080000:2.0:1734018836.032930:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018836.032933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018836.032937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018836.032944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334530944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018836.032945:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334530432:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734018836.032952:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018836.032955:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018836.032957:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018836.032960:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3102 00000100:00080000:2.0:1734018836.032962:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018836.032965:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018836.032969:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018836.032976:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334531072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018836.032988:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334530560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018836.033014:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334530688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018836.033039:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334530816:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018836.033062:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334530944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018836.033088:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334531072:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018837.014742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018838.016735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018839.018717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018839.038761:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734018839.038811:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018839.038833:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018839.038886:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018839.038900:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018839.038911:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018839.038921:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018839.038932:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018839.038938:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:0.0:1734018839.038992:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018836/real 1734018839] req@ffff8800a78a3b80 x1818247334531072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018839.039031:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334531072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018839.039049:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018839.039055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018839.039063:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334531072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018839.039075:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018836/real 1734018839] req@ffff8800a78a1880 x1818247334530560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018839.039093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334530560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018839.039105:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018839.039110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018839.039116:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334530560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018839.039126:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018836/real 1734018839] req@ffff8800a78a2680 x1818247334530688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018839.039142:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334530688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018839.039154:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018839.039157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018839.039163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334530688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018839.039173:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018836/real 1734018839] req@ffff8800a78a0a80 x1818247334530816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018839.039191:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334530816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018839.039204:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018839.039208:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018839.039214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334530816:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018839.039224:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018836/real 1734018839] req@ffff8800a78a1f80 x1818247334530944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018839.039243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334530944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018839.039293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018839.039297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018839.039303:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334530944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018839.039313:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018836/real 1734018839] req@ffff8800a78a3480 x1818247334530304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018839.039332:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334530304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018839.039343:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018839.039347:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018839.039352:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334530304:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018839.039362:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018836/real 1734018839] req@ffff8800a78a0700 x1818247334530432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018839.039379:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334530432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018839.039390:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018839.039394:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018839.039399:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334530432:192.168.201.116@tcp:38:kworker.0 00010000:00010000:3.0:1734018840.020772:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018841.022731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018841.040581:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018841.040591:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018841.040596:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018841.040607:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3107 00000100:00080000:2.0:1734018841.040615:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018841.040623:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018841.040663:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334531200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018841.040696:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018841.040700:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018841.040702:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018841.040706:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3107 00000100:00080000:2.0:1734018841.040709:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018841.040713:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018841.040723:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018841.040735:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334531328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018841.040745:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018841.040748:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018841.040750:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018841.040754:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3107 00000100:00080000:2.0:1734018841.040757:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018841.040760:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018841.040766:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018841.040775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334531456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018841.040785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018841.040788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018841.040790:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018841.040793:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3107 00000100:00080000:2.0:1734018841.040795:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734018841.040797:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334531200:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018841.040798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018841.040802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018841.040810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334531584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018841.040818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018841.040821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018841.040823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018841.040826:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3107 00000100:00080000:2.0:1734018841.040829:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018841.040832:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018841.040836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018841.040843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334531712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018841.040852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018841.040855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018841.040857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018841.040861:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3107 00000800:00000100:0.0:1734018841.040862:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018841.040864:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018841.040867:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018841.040871:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018841.040878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334531840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018841.040886:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018841/real 1734018841] req@ffff8800a78a0700 x1818247334531200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018841.040886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018841.040889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018841.040890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018841.040894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3107 00000100:00080000:2.0:1734018841.040896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018841.040899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018841.040903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018841.040907:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334531200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018841.040911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334531968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018841.040918:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018841.040921:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018841.040926:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334531200:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018841.040933:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334531328:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018841.040951:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018841.040958:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018841/real 1734018841] req@ffff8800a78a3480 x1818247334531328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018841.040969:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334531328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018841.040977:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018841.040979:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018841.040983:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334531328:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018841.041004:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334531456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018841.041017:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018841.041024:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018841/real 1734018841] req@ffff8800a78a1f80 x1818247334531456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018841.041043:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334531456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018841.041052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018841.041054:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018841.041057:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334531456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018841.041063:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334531584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018841.041075:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018841.041081:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018841/real 1734018841] req@ffff8800a78a0a80 x1818247334531584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018841.041090:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334531584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018841.041098:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018841.041100:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018841.041103:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334531584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018841.041109:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334531712:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018841.041122:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018841.041128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018841/real 1734018841] req@ffff8800a78a2680 x1818247334531712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018841.041138:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334531712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018841.041145:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018841.041148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018841.041151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334531712:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018841.041157:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334531840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018841.041169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018841.041175:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018841/real 1734018841] req@ffff8800a78a1880 x1818247334531840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018841.041185:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334531840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018841.041192:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018841.041194:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018841.041197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334531840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018841.041203:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334531968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018841.041215:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018841.041266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018841/real 1734018841] req@ffff8800a78a3b80 x1818247334531968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018841.041277:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334531968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018841.041284:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018841.041286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018841.041289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334531968:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018842.024714:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018843.026704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018844.028703:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018845.030682:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018846.032637:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018846.048576:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018846.048589:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018846.048594:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018846.048607:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3112 00000100:00080000:2.0:1734018846.048618:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018846.048627:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018846.048675:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334532096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018846.048718:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018846.048723:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018846.048724:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018846.048729:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3112 00000100:00080000:2.0:1734018846.048733:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018846.048737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018846.048748:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018846.048761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334532224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018846.048770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018846.048774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018846.048775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018846.048779:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3112 00000100:00080000:2.0:1734018846.048782:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018846.048786:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018846.048791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018846.048800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334532352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018846.048809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018846.048813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018846.048814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018846.048818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3112 00000100:00080000:2.0:1734018846.048820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734018846.048822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334532096:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018846.048825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018846.048829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018846.048838:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334532480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018846.048847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018846.048850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018846.048851:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018846.048855:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3112 00000100:00080000:2.0:1734018846.048857:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018846.048861:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018846.048865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018846.048873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334532608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018846.048880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018846.048883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018846.048885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018846.048889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3112 00000100:00080000:2.0:1734018846.048892:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018846.048895:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018846.048899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018846.048907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334532736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734018846.048911:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018846.048915:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018846.048920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018846.048921:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018846.048925:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3112 00000100:00080000:2.0:1734018846.048937:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734018846.048939:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018846/real 1734018846] req@ffff8800a78a3b80 x1818247334532096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018846.048941:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018846.048946:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018846.048955:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334532864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018846.048961:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334532096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018846.048973:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018846.048976:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018846.048981:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334532096:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018846.048988:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334532224:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018846.049006:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018846.049013:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018846/real 1734018846] req@ffff8800a78a1880 x1818247334532224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018846.049025:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334532224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018846.049033:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018846.049035:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018846.049039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334532224:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018846.049064:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334532352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018846.049078:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018846.049085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018846/real 1734018846] req@ffff8800a78a2680 x1818247334532352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018846.049095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334532352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018846.049104:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018846.049107:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018846.049110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334532352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018846.049115:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334532480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018846.049128:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018846.049174:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018846/real 1734018846] req@ffff8800a78a0a80 x1818247334532480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018846.049185:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334532480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018846.049193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018846.049195:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018846.049199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334532480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018846.049205:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334532608:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018846.049221:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018846.049227:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018846/real 1734018846] req@ffff8800a78a1f80 x1818247334532608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018846.049238:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334532608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018846.049245:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018846.049247:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018846.049250:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334532608:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018846.049257:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334532736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018846.049270:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018846.049276:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018846/real 1734018846] req@ffff8800a78a3480 x1818247334532736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018846.049285:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334532736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018846.049293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018846.049295:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018846.049299:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334532736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018846.049304:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334532864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018846.049316:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018846.049322:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018846/real 1734018846] req@ffff8800a78a0700 x1818247334532864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018846.049332:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334532864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018846.049339:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018846.049342:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018846.049345:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334532864:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018847.034681:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018848.036636:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018849.038653:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018850.040643:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018851.042753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018851.056620:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018851.056632:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018851.056638:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018851.056652:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3117 00000100:00080000:2.0:1734018851.056664:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018851.056674:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018851.056726:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334532992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018851.056765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018851.056769:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018851.056772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018851.056777:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3117 00000100:00080000:2.0:1734018851.056780:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018851.056784:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018851.056795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018851.056807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334533120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018851.056819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018851.056822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018851.056824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018851.056828:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3117 00000100:00080000:2.0:1734018851.056830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018851.056835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018851.056839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018851.056849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334533248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018851.056858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018851.056862:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018851.056863:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018851.056867:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3117 00000100:00080000:2.0:1734018851.056869:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018851.056873:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018851.056877:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018851.056886:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334533376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018851.056896:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334532992:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018851.056896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018851.056899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018851.056900:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018851.056904:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3117 00000100:00080000:2.0:1734018851.056908:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018851.056911:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018851.056916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018851.056925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334533504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018851.056933:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018851.056936:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018851.056937:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018851.056941:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3117 00000100:00080000:2.0:1734018851.056943:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018851.056947:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018851.056951:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018851.056958:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334533632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018851.056966:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1734018851.056970:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018851.056970:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018851.056971:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018851.056975:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3117 00000100:00080000:2.0:1734018851.056978:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018851.056982:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018851.056986:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018851.056995:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018851/real 1734018851] req@ffff8800a78a0700 x1818247334532992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018851.056995:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334533760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018851.057017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334532992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018851.057030:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018851.057034:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018851.057039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334532992:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018851.057095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334533120:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018851.057119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018851.057128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018851/real 1734018851] req@ffff8800a78a3480 x1818247334533120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018851.057140:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334533120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018851.057148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018851.057151:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018851.057154:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334533120:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018851.057161:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334533248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018851.057175:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018851.057182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018851/real 1734018851] req@ffff8800a78a1f80 x1818247334533248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018851.057193:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334533248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018851.057201:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018851.057204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018851.057207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334533248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018851.057236:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334533376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018851.057250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018851.057256:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018851/real 1734018851] req@ffff8800a78a0a80 x1818247334533376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018851.057267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334533376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018851.057274:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018851.057277:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018851.057279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334533376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018851.057285:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334533504:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018851.057298:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018851.057304:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018851/real 1734018851] req@ffff8800a78a2680 x1818247334533504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018851.057314:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334533504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018851.057321:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018851.057324:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018851.057327:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334533504:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018851.057333:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334533632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018851.057349:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018851.057355:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018851/real 1734018851] req@ffff8800a78a1880 x1818247334533632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018851.057366:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334533632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018851.057373:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018851.057375:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018851.057378:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334533632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018851.057384:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334533760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018851.057396:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018851.057402:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018851/real 1734018851] req@ffff8800a78a3b80 x1818247334533760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018851.057412:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334533760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018851.057419:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018851.057422:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018851.057425:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334533760:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018852.044715:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018853.046773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018854.048735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018855.050774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018856.052723:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018856.064596:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018856.064609:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018856.064615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018856.064628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3122 00000100:00080000:2.0:1734018856.064638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018856.064647:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018856.064699:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334533888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018856.064738:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018856.064743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018856.064745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018856.064750:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3122 00000100:00080000:2.0:1734018856.064753:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018856.064757:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018856.064769:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018856.064782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334534016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018856.064791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018856.064794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018856.064796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018856.064801:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3122 00000100:00080000:2.0:1734018856.064803:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018856.064807:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018856.064813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018856.064823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334534144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018856.064833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018856.064837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018856.064838:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018856.064842:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3122 00000100:00080000:2.0:1734018856.064845:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018856.064848:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018856.064852:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018856.064859:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334533888:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018856.064862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334534272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018856.064871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018856.064874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018856.064876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018856.064880:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3122 00000100:00080000:2.0:1734018856.064882:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018856.064885:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018856.064890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018856.064898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334534400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018856.064907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018856.064910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018856.064913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018856.064917:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3122 00000100:00080000:2.0:1734018856.064921:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018856.064924:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018856.064928:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734018856.064931:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018856.064937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334534528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018856.064945:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018856.064948:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018856.064950:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018856.064953:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3122 00000100:00100000:0.0:1734018856.064956:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018856/real 1734018856] req@ffff8800a78a3b80 x1818247334533888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018856.064956:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018856.064960:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018856.064999:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018856.065009:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334534656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018856.065014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334533888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018856.065025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018856.065028:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018856.065034:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334533888:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018856.065043:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334534016:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018856.065064:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018856.065072:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018856/real 1734018856] req@ffff8800a78a1880 x1818247334534016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018856.065083:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334534016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018856.065091:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018856.065094:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018856.065097:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334534016:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018856.065104:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334534144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018856.065118:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018856.065124:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018856/real 1734018856] req@ffff8800a78a2680 x1818247334534144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018856.065134:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334534144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018856.065141:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018856.065145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018856.065148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334534144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018856.065174:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334534272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018856.065187:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018856.065193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018856/real 1734018856] req@ffff8800a78a0a80 x1818247334534272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018856.065204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334534272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018856.065212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018856.065214:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018856.065217:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334534272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018856.065223:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334534400:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018856.065235:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018856.065241:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018856/real 1734018856] req@ffff8800a78a1f80 x1818247334534400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018856.065251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334534400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018856.065259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018856.065261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018856.065264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334534400:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018856.065270:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334534528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018856.065287:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018856.065293:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018856/real 1734018856] req@ffff8800a78a3480 x1818247334534528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018856.065303:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334534528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018856.065310:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018856.065313:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018856.065316:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334534528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018856.065321:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334534656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018856.065334:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018856.065340:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018856/real 1734018856] req@ffff8800a78a0700 x1818247334534656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018856.065350:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334534656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018856.065357:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018856.065359:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018856.065362:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334534656:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018857.054737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018858.056733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018859.058697:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018860.060738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018861.062780:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018861.072606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018861.072618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018861.072624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018861.072637:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3127 00000100:00080000:2.0:1734018861.072648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018861.072657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018861.072708:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334534784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018861.072752:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018861.072756:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018861.072758:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018861.072763:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3127 00000100:00080000:2.0:1734018861.072767:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018861.072770:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018861.072781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018861.072794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334534912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018861.072803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018861.072806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018861.072808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018861.072812:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3127 00000100:00080000:2.0:1734018861.072815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018861.072819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018861.072824:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018861.072833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334535040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018861.072842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018861.072846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018861.072847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018861.072851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3127 00000100:00080000:2.0:1734018861.072854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734018861.072857:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334534784:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018861.072858:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018861.072863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018861.072872:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334535168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018861.072950:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018861.072953:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018861.072955:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018861.072959:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3127 00000100:00080000:2.0:1734018861.072966:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018861.072969:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018861.072975:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1734018861.072980:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018861.072985:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334535296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018861.072994:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018861.072998:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018861.072999:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018861.073003:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3127 00000100:00080000:2.0:1734018861.073006:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734018861.073009:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018861/real 1734018861] req@ffff8800a78a0700 x1818247334534784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018861.073010:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018861.073014:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018861.073022:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334535424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018861.073030:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018861.073034:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018861.073036:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018861.073040:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334534784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018861.073040:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3127 00000100:00080000:2.0:1734018861.073042:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018861.073046:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018861.073050:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734018861.073051:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018861.073054:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018861.073058:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334535552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018861.073059:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334534784:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018861.073067:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334534912:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018861.073085:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018861.073092:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018861/real 1734018861] req@ffff8800a78a3480 x1818247334534912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018861.073103:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334534912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018861.073111:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018861.073114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018861.073117:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334534912:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018861.073143:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334535040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018861.073158:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018861.073164:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018861/real 1734018861] req@ffff8800a78a1f80 x1818247334535040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018861.073174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334535040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018861.073182:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018861.073185:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018861.073189:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334535040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018861.073195:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334535168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018861.073208:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018861.073214:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018861/real 1734018861] req@ffff8800a78a0a80 x1818247334535168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018861.073223:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334535168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018861.073231:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018861.073234:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018861.073237:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334535168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018861.073243:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334535296:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018861.073256:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018861.073262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018861/real 1734018861] req@ffff8800a78a2680 x1818247334535296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018861.073272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334535296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018861.073279:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018861.073281:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018861.073285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334535296:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018861.073290:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334535424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018861.073306:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018861.073313:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018861/real 1734018861] req@ffff8800a78a1880 x1818247334535424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018861.073323:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334535424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018861.073331:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018861.073333:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018861.073336:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334535424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018861.073341:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334535552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018861.073353:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018861.073359:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018861/real 1734018861] req@ffff8800a78a3b80 x1818247334535552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018861.073369:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334535552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018861.073376:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018861.073378:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018861.073381:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334535552:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018862.064768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018863.066737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018864.068769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018865.070765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018866.072715:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018866.080594:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018866.080604:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018866.080609:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018866.080619:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3132 00000100:00080000:2.0:1734018866.080628:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018866.080636:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018866.080674:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334535680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018866.080707:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018866.080712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018866.080714:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018866.080718:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3132 00000100:00080000:2.0:1734018866.080721:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018866.080725:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018866.080735:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018866.080746:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334535808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018866.080757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018866.080761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018866.080762:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018866.080766:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3132 00000100:00080000:2.0:1734018866.080769:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018866.080773:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018866.080777:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018866.080786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334535936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018866.080835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018866.080840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734018866.080841:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334535680:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018866.080841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018866.080846:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3132 00000100:00080000:2.0:1734018866.080849:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018866.080853:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018866.080858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018866.080869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334536064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018866.080878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018866.080881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018866.080883:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018866.080886:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3132 00000100:00080000:2.0:1734018866.080891:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018866.080894:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018866.080898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018866.080906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334536192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734018866.080913:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018866.080914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018866.080916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018866.080918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018866.080921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3132 00000100:00080000:2.0:1734018866.080924:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018866.080927:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018866.080931:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018866.080938:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018866/real 1734018866] req@ffff8800a78a3b80 x1818247334535680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018866.080939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334536320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018866.080946:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018866.080950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018866.080952:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018866.080955:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3132 00000100:00080000:2.0:1734018866.080958:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018866.080961:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018866.080965:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018866.080966:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334535680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018866.080972:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334536448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018866.080976:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018866.080979:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018866.080984:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334535680:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018866.080992:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334535808:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018866.081009:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018866.081016:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018866/real 1734018866] req@ffff8800a78a1880 x1818247334535808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018866.081027:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334535808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018866.081035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018866.081037:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018866.081041:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334535808:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018866.081063:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334535936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018866.081077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018866.081083:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018866/real 1734018866] req@ffff8800a78a2680 x1818247334535936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018866.081093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334535936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018866.081101:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018866.081158:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018866.081163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334535936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018866.081169:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334536064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018866.081185:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018866.081193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018866/real 1734018866] req@ffff8800a78a0a80 x1818247334536064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018866.081203:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334536064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018866.081211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018866.081213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018866.081216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334536064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018866.081221:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334536192:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018866.081234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018866.081240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018866/real 1734018866] req@ffff8800a78a1f80 x1818247334536192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018866.081249:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334536192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018866.081256:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018866.081258:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018866.081262:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334536192:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018866.081267:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334536320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018866.081283:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018866.081289:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018866/real 1734018866] req@ffff8800a78a3480 x1818247334536320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018866.081299:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334536320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018866.081306:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018866.081309:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018866.081312:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334536320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018866.081317:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334536448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018866.081329:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018866.081335:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018866/real 1734018866] req@ffff8800a78a0700 x1818247334536448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018866.081344:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334536448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018866.081351:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018866.081353:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018866.081356:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334536448:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018867.074720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018868.076678:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018869.078725:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018870.080762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018871.082804:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018871.088602:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018871.088611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018871.088616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018871.088627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3137 00000100:00080000:2.0:1734018871.088636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018871.088644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018871.088684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334536576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018871.088739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018871.088744:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018871.088746:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018871.088751:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3137 00000100:00080000:2.0:1734018871.088754:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018871.088758:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018871.088768:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018871.088781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334536704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018871.088791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018871.088795:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018871.088797:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018871.088801:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3137 00000100:00080000:2.0:1734018871.088804:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018871.088808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018871.088813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018871.088823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334536832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018871.088833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018871.088836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018871.088838:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018871.088842:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3137 00000100:00080000:2.0:1734018871.088844:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018871.088847:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018871.088852:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018871.088853:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334536576:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018871.088860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334536960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018871.088869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018871.088872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018871.088873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018871.088877:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3137 00000100:00080000:2.0:1734018871.088879:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018871.088882:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018871.088886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018871.088894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334537088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018871.088902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018871.088904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018871.088907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018871.088910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3137 00000100:00080000:2.0:1734018871.088913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018871.088916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018871.088920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018871.088928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334537216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018871.088935:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018871.088938:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018871.088940:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018871.088943:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3137 00000100:00080000:2.0:1734018871.088946:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734018871.088947:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018871.088949:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018871.088953:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018871.088961:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334537344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018871.088976:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018871/real 1734018871] req@ffff8800a78a0700 x1818247334536576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018871.089008:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334536576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018871.089020:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018871.089024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018871.089029:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334536576:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018871.089038:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334536704:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018871.089057:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018871.089065:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018871/real 1734018871] req@ffff8800a78a3480 x1818247334536704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018871.089076:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334536704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018871.089084:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018871.089087:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018871.089090:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334536704:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018871.089119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334536832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018871.089133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018871.089140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018871/real 1734018871] req@ffff8800a78a1f80 x1818247334536832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018871.089150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334536832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018871.089158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018871.089162:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018871.089165:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334536832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018871.089172:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334536960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018871.089186:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018871.089192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018871/real 1734018871] req@ffff8800a78a0a80 x1818247334536960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018871.089202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334536960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018871.089209:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018871.089211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018871.089214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334536960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018871.089220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334537088:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018871.089234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018871.089240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018871/real 1734018871] req@ffff8800a78a2680 x1818247334537088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018871.089250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334537088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018871.089258:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018871.089260:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018871.089263:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334537088:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018871.089269:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334537216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018871.089286:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018871.089292:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018871/real 1734018871] req@ffff8800a78a1880 x1818247334537216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018871.089303:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334537216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018871.089310:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018871.089312:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018871.089315:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334537216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018871.089321:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334537344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018871.089334:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018871.089340:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018871/real 1734018871] req@ffff8800a78a3b80 x1818247334537344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018871.089350:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334537344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018871.089358:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018871.089360:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018871.089363:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334537344:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018872.084778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018873.086756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018874.088825:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018875.090781:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018876.092810:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018876.096601:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018876.096616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018876.096620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018876.096667:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3142 00000100:00080000:2.0:1734018876.096676:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018876.096684:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018876.096722:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334537472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018876.096765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018876.096769:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018876.096771:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018876.096775:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3142 00000100:00080000:2.0:1734018876.096778:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018876.096782:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018876.096792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018876.096804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334537600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018876.096813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018876.096816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018876.096818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018876.096822:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3142 00000100:00080000:2.0:1734018876.096825:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018876.096829:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018876.096834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018876.096843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334537728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018876.096852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018876.096855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018876.096856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018876.096860:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3142 00000100:00080000:2.0:1734018876.096862:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018876.096866:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018876.096870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018876.096878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334537856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018876.096888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018876.096891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018876.096892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018876.096893:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334537472:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018876.096896:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3142 00000100:00080000:2.0:1734018876.096898:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018876.096902:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018876.096906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018876.096914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334537984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018876.096921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018876.096924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018876.096926:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018876.096929:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3142 00000100:00080000:2.0:1734018876.096931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018876.096935:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018876.096939:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018876.096946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334538112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018876.096954:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018876.096958:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018876.096960:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018876.096963:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3142 00000100:00080000:2.0:1734018876.096967:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018876.096970:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018876.096974:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018876.096981:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334538240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734018876.096987:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018876.097017:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018876/real 1734018876] req@ffff8800a78a3b80 x1818247334537472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018876.097048:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334537472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018876.097059:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018876.097063:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018876.097069:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334537472:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018876.097077:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334537600:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018876.097095:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018876.097103:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018876/real 1734018876] req@ffff8800a78a1880 x1818247334537600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018876.097114:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334537600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018876.097122:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018876.097125:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018876.097128:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334537600:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018876.097134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334537728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018876.097149:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018876.097155:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018876/real 1734018876] req@ffff8800a78a2680 x1818247334537728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018876.097165:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334537728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018876.097173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018876.097176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018876.097180:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334537728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018876.097211:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334537856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018876.097225:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018876.097231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018876/real 1734018876] req@ffff8800a78a0a80 x1818247334537856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018876.097242:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334537856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018876.097250:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018876.097252:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018876.097255:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334537856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018876.097261:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334537984:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018876.097274:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018876.097280:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018876/real 1734018876] req@ffff8800a78a1f80 x1818247334537984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018876.097290:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334537984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018876.097298:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018876.097300:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018876.097303:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334537984:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018876.097309:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334538112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018876.097326:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018876.097333:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018876/real 1734018876] req@ffff8800a78a3480 x1818247334538112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018876.097344:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334538112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018876.097351:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018876.097353:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018876.097356:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334538112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018876.097362:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334538240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018876.097375:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018876.097381:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018876/real 1734018876] req@ffff8800a78a0700 x1818247334538240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018876.097391:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334538240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018876.097398:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018876.097400:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018876.097403:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334538240:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018877.094753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018878.096813:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018879.098774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018880.100770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018881.102782:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018881.104660:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018881.104671:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018881.104677:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018881.104690:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3147 00000100:00080000:2.0:1734018881.104701:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018881.104711:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018881.104762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334538368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018881.104798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018881.104802:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018881.104805:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018881.104810:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3147 00000100:00080000:2.0:1734018881.104813:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018881.104817:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018881.104827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018881.104840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334538496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018881.104852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018881.104855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018881.104857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018881.104861:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3147 00000100:00080000:2.0:1734018881.104863:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018881.104867:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018881.104872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018881.104881:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334538624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018881.104890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018881.104893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018881.104895:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018881.104898:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3147 00000100:00080000:2.0:1734018881.104900:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018881.104904:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734018881.104908:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334538368:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734018881.104909:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018881.104917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334538752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018881.104927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018881.104930:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018881.104931:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018881.104935:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3147 00000100:00080000:2.0:1734018881.104938:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018881.104942:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018881.104946:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018881.104954:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334538880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018881.104962:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018881.104965:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018881.104967:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018881.104970:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3147 00000100:00080000:2.0:1734018881.104973:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018881.104977:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018881.104981:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018881.104989:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334539008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734018881.104994:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018881.104996:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018881.105000:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018881.105001:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018881.105005:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3147 00000100:00080000:2.0:1734018881.105008:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018881.105011:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018881.105015:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018881.105023:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018881/real 1734018881] req@ffff8800a78a0700 x1818247334538368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018881.105024:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334539136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018881.105054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334538368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018881.105065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018881.105069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018881.105074:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334538368:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018881.105082:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334538496:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018881.105100:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018881.105107:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018881/real 1734018881] req@ffff8800a78a3480 x1818247334538496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018881.105118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334538496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018881.105126:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018881.105129:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018881.105132:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334538496:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018881.105138:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334538624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018881.105152:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018881.105158:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018881/real 1734018881] req@ffff8800a78a1f80 x1818247334538624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018881.105168:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334538624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018881.105176:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018881.105179:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018881.105182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334538624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018881.105212:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334538752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018881.105226:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018881.105232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018881/real 1734018881] req@ffff8800a78a0a80 x1818247334538752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018881.105243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334538752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018881.105250:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018881.105252:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018881.105255:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334538752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018881.105260:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334538880:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018881.105273:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018881.105279:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018881/real 1734018881] req@ffff8800a78a2680 x1818247334538880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018881.105289:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334538880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018881.105296:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018881.105298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018881.105301:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334538880:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018881.105307:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334539008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018881.105322:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018881.105328:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018881/real 1734018881] req@ffff8800a78a1880 x1818247334539008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018881.105338:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334539008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018881.105346:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018881.105348:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018881.105351:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334539008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018881.105356:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334539136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018881.105369:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018881.105375:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018881/real 1734018881] req@ffff8800a78a3b80 x1818247334539136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018881.105385:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334539136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018881.105392:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018881.105394:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018881.105397:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334539136:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018882.104809:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018883.106788:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018884.108779:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018885.110737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018886.112590:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018886.112602:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018886.112606:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018886.112618:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3152 00000100:00080000:2.0:1734018886.112626:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018886.112633:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018886.112672:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334539264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018886.112707:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018886.112711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018886.112713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018886.112717:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3152 00000100:00080000:2.0:1734018886.112720:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018886.112724:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018886.112734:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00010000:00010000:3.0:1734018886.112742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734018886.112746:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334539392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018886.112748:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334539264:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018886.112756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018886.112760:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018886.112761:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018886.112766:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3152 00000100:00080000:2.0:1734018886.112768:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018886.112772:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018886.112778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018886.112787:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334539520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018886.112796:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018886.112799:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018886.112800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018886.112804:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3152 00000100:00080000:2.0:1734018886.112807:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018886.112810:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018886.112814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000800:00000100:0.0:1734018886.112816:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018886.112822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334539648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018886.112830:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018886.112833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018886.112835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018886.112839:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3152 00000100:00100000:0.0:1734018886.112840:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018886/real 1734018886] req@ffff8800a78a3b80 x1818247334539264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018886.112842:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018886.112845:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018886.112849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018886.112856:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334539776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018886.112864:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018886.112867:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1734018886.112868:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334539264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018886.112869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018886.112873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3152 00000100:00080000:2.0:1734018886.112876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018886.112878:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018886.112879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:0.0:1734018886.112881:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000008:00100000:2.0:1734018886.112883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018886.112886:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334539264:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018886.112891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334539904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018886.112899:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018886.112902:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018886.112903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018886.112906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3152 00000100:00100000:0.0:1734018886.112909:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334539392:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734018886.112909:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018886.112912:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018886.112916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018886.112923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334540032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734018886.112926:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018886.112933:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018886/real 1734018886] req@ffff8800a78a1880 x1818247334539392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018886.112945:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334539392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018886.112952:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018886.112955:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018886.112959:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334539392:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018886.112965:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334539520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018886.112978:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018886.112984:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018886/real 1734018886] req@ffff8800a78a2680 x1818247334539520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018886.112995:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334539520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018886.113003:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018886.113006:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018886.113009:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334539520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018886.113015:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334539648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018886.113027:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018886.113033:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018886/real 1734018886] req@ffff8800a78a0a80 x1818247334539648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018886.113043:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334539648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018886.113050:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018886.113053:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018886.113055:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334539648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018886.113061:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334539776:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018886.113073:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018886.113079:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018886/real 1734018886] req@ffff8800a78a1f80 x1818247334539776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018886.113089:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334539776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018886.113096:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018886.113098:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018886.113101:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334539776:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018886.113107:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334539904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018886.113122:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018886.113128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018886/real 1734018886] req@ffff8800a78a3480 x1818247334539904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018886.113139:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334539904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018886.113146:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018886.113148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018886.113151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334539904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018886.113167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334540032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018886.113179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018886.113186:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018886/real 1734018886] req@ffff8800a78a0700 x1818247334540032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018886.113196:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334540032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018886.113202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018886.113205:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018886.113208:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334540032:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018887.114774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018888.116807:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018889.118760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018890.120724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018891.120632:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018891.120643:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018891.120647:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018891.120657:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3157 00000100:00080000:2.0:1734018891.120666:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018891.120674:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018891.120712:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334540160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018891.120779:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018891.120783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018891.120785:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018891.120790:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3157 00000100:00080000:2.0:1734018891.120794:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018891.120798:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018891.120808:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018891.120820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334540288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018891.120830:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018891.120833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018891.120835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018891.120838:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3157 00000100:00080000:2.0:1734018891.120841:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018891.120845:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018891.120849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018891.120858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334540416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018891.120861:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334540160:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018891.120867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018891.120870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018891.120872:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018891.120875:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3157 00000100:00080000:2.0:1734018891.120878:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018891.120882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018891.120886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018891.120894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334540544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018891.120902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018891.120905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018891.120907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018891.120910:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3157 00000100:00080000:2.0:1734018891.120913:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018891.120916:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018891.120920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1734018891.120928:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018891.120928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334540672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018891.120935:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018891.120938:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018891.120940:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018891.120943:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3157 00000100:00080000:2.0:1734018891.120946:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018891.120949:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734018891.120951:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018891/real 1734018891] req@ffff8800a78a0700 x1818247334540160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018891.120953:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018891.120961:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334540800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018891.120969:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018891.120972:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018891.120974:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018891.120977:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3157 00000100:00100000:0.0:1734018891.120979:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334540160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018891.120980:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018891.120983:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018891.120987:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734018891.120990:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018891.120993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018891.120994:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334540928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018891.120998:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334540160:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018891.121005:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334540288:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018891.121023:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018891.121030:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018891/real 1734018891] req@ffff8800a78a3480 x1818247334540288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018891.121041:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334540288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018891.121049:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018891.121052:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018891.121055:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334540288:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018891.121078:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334540416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018891.121091:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018891.121098:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018891/real 1734018891] req@ffff8800a78a1f80 x1818247334540416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018891.121108:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334540416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018891.121116:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018891.121119:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018891.121123:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334540416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018891.121129:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334540544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018891.121142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018891.121148:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018891/real 1734018891] req@ffff8800a78a0a80 x1818247334540544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018891.121158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334540544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018891.121165:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018891.121167:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018891.121170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334540544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018891.121176:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334540672:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018891.121188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018891.121194:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018891/real 1734018891] req@ffff8800a78a2680 x1818247334540672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018891.121204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334540672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018891.121211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018891.121213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018891.121216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334540672:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018891.121221:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334540800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018891.121237:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018891.121244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018891/real 1734018891] req@ffff8800a78a1880 x1818247334540800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018891.121254:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334540800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018891.121261:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018891.121263:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018891.121266:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334540800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018891.121271:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334540928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018891.121283:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018891.121289:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018891/real 1734018891] req@ffff8800a78a3b80 x1818247334540928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018891.121299:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334540928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018891.121306:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018891.121308:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018891.121311:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334540928:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018891.122742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018892.124797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018893.126808:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018894.128793:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018895.130722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018896.128582:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018896.128593:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018896.128598:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018896.128609:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3162 00000100:00080000:2.0:1734018896.128618:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018896.128626:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018896.128664:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334541056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018896.128696:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018896.128700:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018896.128702:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018896.128707:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3162 00000100:00080000:2.0:1734018896.128710:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018896.128713:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018896.128722:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018896.128734:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334541184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018896.128745:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018896.128748:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018896.128750:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018896.128754:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3162 00000100:00080000:2.0:1734018896.128756:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018896.128760:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018896.128764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018896.128772:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334541312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018896.128781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018896.128784:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018896.128786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018896.128789:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3162 00000100:00080000:2.0:1734018896.128791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018896.128795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734018896.128799:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334541056:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734018896.128799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018896.128807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334541440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018896.128816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018896.128819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018896.128820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018896.128824:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3162 00000100:00080000:2.0:1734018896.128827:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018896.128830:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018896.128834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018896.128841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334541568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018896.128849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018896.128852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018896.128854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018896.128857:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3162 00000100:00080000:2.0:1734018896.128860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018896.128863:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:0.0:1734018896.128865:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734018896.128867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018896.128875:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334541696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018896.128882:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018896.128886:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734018896.128887:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018896/real 1734018896] req@ffff8800a78a3b80 x1818247334541056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018896.128887:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018896.128890:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3162 00000100:00080000:2.0:1734018896.128893:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018896.128896:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018896.128900:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018896.128907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334541824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018896.128916:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334541056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018896.128925:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018896.128929:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018896.128933:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334541056:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018896.128941:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334541184:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018896.128957:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018896.128964:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018896/real 1734018896] req@ffff8800a78a1880 x1818247334541184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018896.128975:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334541184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018896.128983:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018896.128985:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018896.128989:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334541184:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018896.128994:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334541312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018896.129007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018896.129014:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018896/real 1734018896] req@ffff8800a78a2680 x1818247334541312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018896.129024:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334541312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018896.129031:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018896.129034:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018896.129038:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334541312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018896.129062:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334541440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018896.129075:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018896.129081:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018896/real 1734018896] req@ffff8800a78a0a80 x1818247334541440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018896.129092:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334541440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018896.129099:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018896.129101:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018896.129104:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334541440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018896.129110:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334541568:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018896.129122:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018896.129128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018896/real 1734018896] req@ffff8800a78a1f80 x1818247334541568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018896.129138:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334541568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018896.129145:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018896.129147:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018896.129150:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334541568:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018896.129155:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334541696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018896.129170:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018896.129177:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018896/real 1734018896] req@ffff8800a78a3480 x1818247334541696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018896.129187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334541696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018896.129194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018896.129196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018896.129199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334541696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018896.129204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334541824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018896.129217:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018896.129223:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018896/real 1734018896] req@ffff8800a78a0700 x1818247334541824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018896.129232:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334541824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018896.129239:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018896.129242:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018896.129244:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334541824:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018896.132784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018897.134792:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018898.136801:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018899.138795:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018900.140740:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018901.136608:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018901.136619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018901.136624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018901.136635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3167 00000100:00080000:2.0:1734018901.136644:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018901.136652:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018901.136692:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334541952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018901.136729:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018901.136733:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018901.136736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018901.136740:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3167 00000100:00080000:2.0:1734018901.136744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018901.136747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018901.136757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018901.136769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334542080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018901.136778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018901.136781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018901.136783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018901.136787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3167 00000100:00080000:2.0:1734018901.136790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018901.136793:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018901.136799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018901.136809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334542208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018901.136818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018901.136821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018901.136823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018901.136826:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3167 00000100:00080000:2.0:1734018901.136829:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018901.136832:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018901.136836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018901.136843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334542336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018901.136853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018901.136856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018901.136857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018901.136861:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3167 00000100:00100000:0.0:1734018901.136863:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334541952:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018901.136863:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018901.136866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018901.136870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018901.136878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334542464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018901.136886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018901.136889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018901.136891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018901.136895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3167 00000100:00080000:2.0:1734018901.136898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018901.136901:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018901.136905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018901.136912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334542592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018901.136920:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018901.136923:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018901.136924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018901.136927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3167 00000100:00080000:2.0:1734018901.136932:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018901.136935:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018901.136939:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018901.136946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334542720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018901.136994:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334542080:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018901.137020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334542208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018901.137049:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334542336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018901.137067:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334542464:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018901.137084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334542592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018901.137100:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334542720:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018901.142750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018902.144739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018903.146744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018904.142734:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734018904.142784:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018904.142804:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018904.142851:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018904.142859:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018904.142867:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018904.142874:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018904.142881:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018904.142886:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734018904.142933:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018901/real 1734018904] req@ffff8800a78a0a80 x1818247334542336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018904.142971:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334542336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018904.142983:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018904.142988:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018904.142993:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334542336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018904.143004:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018901/real 1734018904] req@ffff8800a78a2680 x1818247334542464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018904.143017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334542464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018904.143026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018904.143028:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018904.143032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334542464:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018904.143040:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018901/real 1734018904] req@ffff8800a78a1880 x1818247334542592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018904.143052:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334542592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018904.143059:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018904.143062:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018904.143065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334542592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018904.143073:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018901/real 1734018904] req@ffff8800a78a3b80 x1818247334542720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018904.143085:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334542720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018904.143092:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018904.143095:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018904.143099:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334542720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018904.143109:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018901/real 1734018904] req@ffff8800a78a0700 x1818247334541952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018904.143130:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334541952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018904.143138:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018904.143141:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018904.143144:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334541952:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018904.145310:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018901/real 1734018904] req@ffff8800a78a3480 x1818247334542080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018904.145361:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334542080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018904.145381:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018904.145386:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018904.145392:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334542080:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018904.145400:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018901/real 1734018904] req@ffff8800a78a1f80 x1818247334542208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018904.145413:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334542208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018904.145420:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018904.145423:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018904.145427:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334542208:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018904.149978:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018905.152960:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018906.144572:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018906.144584:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018906.144588:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018906.144598:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3172 00000100:00080000:2.0:1734018906.144607:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018906.144615:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018906.144691:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334542848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018906.144731:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018906.144735:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018906.144737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018906.144741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3172 00000100:00080000:2.0:1734018906.144744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018906.144748:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018906.144758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018906.144770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334542976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018906.144780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018906.144783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018906.144784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018906.144789:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3172 00000100:00080000:2.0:1734018906.144792:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018906.144795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018906.144799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018906.144808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334543104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018906.144817:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018906.144820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018906.144822:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018906.144826:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3172 00000100:00080000:2.0:1734018906.144828:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018906.144832:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018906.144836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018906.144844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334543232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018906.144852:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334542848:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018906.144853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018906.144856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018906.144858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018906.144861:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3172 00000100:00080000:2.0:1734018906.144864:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018906.144867:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018906.144870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018906.144878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334543360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018906.144886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018906.144889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018906.144890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018906.144893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3172 00000100:00080000:2.0:1734018906.144897:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018906.144900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018906.144904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018906.144911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334543488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018906.144919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734018906.144920:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018906.144923:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018906.144924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018906.144928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3172 00000100:00080000:2.0:1734018906.144931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018906.144934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018906.144938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018906.144941:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018906/real 1734018906] req@ffff8800a78a1f80 x1818247334542848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018906.144945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334543616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018906.144960:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334542848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018906.144971:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018906.144974:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018906.144979:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334542848:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018906.144986:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334542976:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018906.145003:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018906.145010:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018906/real 1734018906] req@ffff8800a78a3480 x1818247334542976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018906.145021:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334542976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018906.145029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018906.145032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018906.145035:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334542976:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018906.145041:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334543104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018906.145055:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018906.145061:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018906/real 1734018906] req@ffff8800a78a0700 x1818247334543104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018906.145071:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334543104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018906.145079:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018906.145082:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018906.145085:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334543104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018906.145108:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334543232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018906.145156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018906.145164:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018906/real 1734018906] req@ffff8800a78a3b80 x1818247334543232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018906.145174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334543232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018906.145182:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018906.145185:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018906.145188:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334543232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018906.145194:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334543360:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018906.145206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018906.145212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018906/real 1734018906] req@ffff8800a78a1880 x1818247334543360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018906.145222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334543360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018906.145230:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018906.145232:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018906.145235:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334543360:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018906.145242:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334543488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018906.145254:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018906.145260:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018906/real 1734018906] req@ffff8800a78a2680 x1818247334543488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018906.145270:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334543488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018906.145277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018906.145279:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018906.145282:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334543488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018906.145287:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334543616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018906.145299:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018906.145306:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018906/real 1734018906] req@ffff8800a78a0a80 x1818247334543616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018906.145315:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334543616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018906.145322:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018906.145324:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018906.145327:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334543616:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018906.154768:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018907.156729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018908.158692:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018909.160755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018910.162763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018911.152605:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018911.152616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018911.152621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018911.152632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3177 00000100:00080000:2.0:1734018911.152641:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018911.152648:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018911.152686:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334543744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018911.152719:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018911.152723:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018911.152725:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018911.152730:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3177 00000100:00080000:2.0:1734018911.152733:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018911.152736:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018911.152746:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018911.152757:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334543872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018911.152768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018911.152771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018911.152773:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018911.152777:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3177 00000100:00080000:2.0:1734018911.152779:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018911.152782:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018911.152787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018911.152795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334544000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018911.152804:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018911.152807:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018911.152808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018911.152812:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3177 00000100:00080000:2.0:1734018911.152814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018911.152817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018911.152821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018911.152830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334544128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018911.152837:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334543744:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018911.152839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018911.152842:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018911.152844:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018911.152847:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3177 00000100:00080000:2.0:1734018911.152850:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018911.152853:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018911.152858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018911.152865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334544256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018911.152873:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018911.152876:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018911.152877:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018911.152881:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3177 00000100:00080000:2.0:1734018911.152883:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018911.152886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018911.152890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018911.152898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334544384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018911.152906:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018911.152906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018911.152909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018911.152910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018911.152914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3177 00000100:00080000:2.0:1734018911.152916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018911.152919:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018911.152923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018911.152927:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018911/real 1734018911] req@ffff8800a78a0a80 x1818247334543744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018911.152931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334544512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018911.152948:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334543744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018911.152959:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018911.152962:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018911.152967:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334543744:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018911.152974:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334543872:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018911.152990:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018911.152997:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018911/real 1734018911] req@ffff8800a78a2680 x1818247334543872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018911.153009:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334543872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018911.153016:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018911.153019:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018911.153022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334543872:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018911.153059:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334544000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018911.153077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018911.153085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018911/real 1734018911] req@ffff8800a78a1880 x1818247334544000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018911.153095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334544000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018911.153103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018911.153106:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018911.153110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334544000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018911.153135:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334544128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018911.153148:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018911.153155:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018911/real 1734018911] req@ffff8800a78a3b80 x1818247334544128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018911.153165:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334544128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018911.153173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018911.153175:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018911.153178:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334544128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018911.153183:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334544256:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018911.153195:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018911.153201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018911/real 1734018911] req@ffff8800a78a0700 x1818247334544256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018911.153211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334544256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018911.153218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018911.153220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018911.153223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334544256:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018911.153228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334544384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018911.153243:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018911.153250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018911/real 1734018911] req@ffff8800a78a3480 x1818247334544384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018911.153260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334544384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018911.153267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018911.153269:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018911.153272:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334544384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018911.153277:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334544512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018911.153289:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018911.153295:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018911/real 1734018911] req@ffff8800a78a1f80 x1818247334544512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018911.153305:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334544512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018911.153312:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018911.153314:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018911.153317:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334544512:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018911.164764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018912.166781:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018913.168774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018914.170809:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018915.172739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018916.160593:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018916.160608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018916.160612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018916.160623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3182 00000100:00080000:2.0:1734018916.160631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018916.160639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018916.160677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334544640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018916.160710:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018916.160714:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018916.160716:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018916.160721:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3182 00000100:00080000:2.0:1734018916.160723:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018916.160727:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018916.160737:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018916.160749:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334544768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018916.160758:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018916.160762:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018916.160763:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018916.160767:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3182 00000100:00080000:2.0:1734018916.160770:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018916.160773:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018916.160779:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018916.160787:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334544896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018916.160796:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018916.160799:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018916.160801:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018916.160804:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3182 00000100:00080000:2.0:1734018916.160807:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018916.160810:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018916.160814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018916.160822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334544640:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018916.160822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334545024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018916.160831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018916.160834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018916.160835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018916.160838:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3182 00000100:00080000:2.0:1734018916.160841:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018916.160844:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018916.160848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018916.160855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334545152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018916.160863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018916.160866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018916.160869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018916.160872:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3182 00000100:00080000:2.0:1734018916.160877:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018916.160880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018916.160884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018916.160887:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018916.160892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334545280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018916.160900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018916.160903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018916.160904:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018916.160907:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3182 00000100:00100000:1.0:1734018916.160908:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018916/real 1734018916] req@ffff8800a78a1f80 x1818247334544640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018916.160910:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018916.160914:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018916.160918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018916.160925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334545408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018916.160936:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334544640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018916.160980:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018916.160983:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018916.160989:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334544640:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018916.160997:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334544768:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018916.161018:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018916.161026:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018916/real 1734018916] req@ffff8800a78a3480 x1818247334544768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018916.161037:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334544768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018916.161045:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018916.161048:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018916.161051:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334544768:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018916.161057:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334544896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018916.161070:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018916.161076:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018916/real 1734018916] req@ffff8800a78a0700 x1818247334544896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018916.161087:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334544896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018916.161094:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018916.161097:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018916.161100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334544896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018916.161127:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334545024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018916.161140:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018916.161146:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018916/real 1734018916] req@ffff8800a78a3b80 x1818247334545024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018916.161157:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334545024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018916.161164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018916.161166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018916.161169:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334545024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018916.161175:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334545152:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018916.161187:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018916.161193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018916/real 1734018916] req@ffff8800a78a1880 x1818247334545152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018916.161202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334545152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018916.161210:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018916.161212:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018916.161215:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334545152:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018916.161220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334545280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018916.161235:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018916.161242:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018916/real 1734018916] req@ffff8800a78a2680 x1818247334545280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018916.161252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334545280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018916.161259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018916.161261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018916.161264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334545280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018916.161269:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334545408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018916.161281:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018916.161287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018916/real 1734018916] req@ffff8800a78a0a80 x1818247334545408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018916.161297:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334545408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018916.161304:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018916.161306:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018916.161309:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334545408:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018916.174784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018917.176816:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018918.178772:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018919.180744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018920.182773:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018921.168599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018921.168611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018921.168615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018921.168626:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3187 00000100:00080000:2.0:1734018921.168635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018921.168642:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018921.168734:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334545536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018921.168775:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018921.168779:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018921.168780:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018921.168785:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3187 00000100:00080000:2.0:1734018921.168789:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018921.168792:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018921.168802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018921.168814:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334545664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018921.168824:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018921.168827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018921.168828:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018921.168832:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3187 00000100:00080000:2.0:1734018921.168835:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018921.168838:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018921.168842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018921.168851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334545792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018921.168888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018921.168891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018921.168893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018921.168896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3187 00000100:00080000:2.0:1734018921.168899:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018921.168903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018921.168908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018921.168911:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334545536:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018921.168918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334545920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018921.168927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018921.168930:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018921.168932:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018921.168935:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3187 00000100:00080000:2.0:1734018921.168937:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018921.168940:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018921.168944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018921.168952:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334546048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018921.168960:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018921.168963:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018921.168964:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018921.168968:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3187 00000100:00080000:2.0:1734018921.168970:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018921.168974:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018921.168976:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734018921.168978:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018921.168985:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334546176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018921.168993:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018921.168997:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018921.168998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018921/real 1734018921] req@ffff8800a78a0a80 x1818247334545536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018921.168999:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018921.169002:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3187 00000100:00080000:2.0:1734018921.169005:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018921.169008:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018921.169012:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018921.169019:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334546304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018921.169024:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334545536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018921.169035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018921.169038:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018921.169042:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334545536:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018921.169049:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334545664:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018921.169067:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018921.169074:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018921/real 1734018921] req@ffff8800a78a2680 x1818247334545664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018921.169085:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334545664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018921.169093:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018921.169095:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018921.169098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334545664:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018921.169104:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334545792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018921.169117:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018921.169123:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018921/real 1734018921] req@ffff8800a78a1880 x1818247334545792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018921.169133:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334545792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018921.169140:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018921.169144:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018921.169147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334545792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018921.169202:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334545920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018921.169220:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018921.169228:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018921/real 1734018921] req@ffff8800a78a3b80 x1818247334545920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018921.169239:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334545920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018921.169246:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018921.169248:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018921.169251:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334545920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018921.169256:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334546048:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018921.169269:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018921.169275:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018921/real 1734018921] req@ffff8800a78a0700 x1818247334546048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018921.169285:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334546048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018921.169293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018921.169295:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018921.169298:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334546048:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018921.169303:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334546176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018921.169318:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018921.169325:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018921/real 1734018921] req@ffff8800a78a3480 x1818247334546176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018921.169335:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334546176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018921.169342:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018921.169344:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018921.169347:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334546176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018921.169352:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334546304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018921.169364:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018921.169370:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018921/real 1734018921] req@ffff8800a78a1f80 x1818247334546304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018921.169380:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334546304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018921.169387:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018921.169389:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018921.169392:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334546304:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018921.184768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018922.186745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018923.188742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018924.190750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018925.192728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018926.176607:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018926.176619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018926.176623:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018926.176633:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3192 00000100:00080000:2.0:1734018926.176642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018926.176650:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018926.176687:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334546432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018926.176721:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018926.176725:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018926.176727:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018926.176732:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3192 00000100:00080000:2.0:1734018926.176735:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018926.176738:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018926.176748:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018926.176759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334546560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018926.176770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018926.176815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018926.176817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018926.176821:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3192 00000100:00080000:2.0:1734018926.176824:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018926.176828:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018926.176833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018926.176843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334546688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018926.176854:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018926.176856:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334546432:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018926.176857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018926.176859:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018926.176862:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3192 00000100:00080000:2.0:1734018926.176865:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018926.176868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018926.176872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018926.176880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334546816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018926.176888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018926.176891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018926.176893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018926.176896:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3192 00000100:00080000:2.0:1734018926.176900:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018926.176903:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018926.176907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018926.176915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334546944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018926.176921:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018926.176923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018926.176926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018926.176927:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018926.176931:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3192 00000100:00080000:2.0:1734018926.176934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018926.176937:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018926.176941:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018926.176944:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018926/real 1734018926] req@ffff8800a78a1f80 x1818247334546432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018926.176948:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334547072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018926.176956:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018926.176959:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018926.176961:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018926.176964:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3192 00000100:00080000:2.0:1734018926.176967:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018926.176970:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018926.176972:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334546432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018926.176973:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018926.176981:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334547200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018926.176982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018926.176985:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018926.176990:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334546432:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018926.176997:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334546560:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018926.177014:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018926.177021:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018926/real 1734018926] req@ffff8800a78a3480 x1818247334546560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018926.177032:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334546560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018926.177040:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018926.177042:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018926.177046:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334546560:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018926.177068:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334546688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018926.177082:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018926.177088:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018926/real 1734018926] req@ffff8800a78a0700 x1818247334546688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018926.177099:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334546688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018926.177106:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018926.177109:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018926.177112:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334546688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018926.177119:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334546816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018926.177131:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018926.177137:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018926/real 1734018926] req@ffff8800a78a3b80 x1818247334546816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018926.177147:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334546816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018926.177154:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018926.177156:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018926.177159:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334546816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018926.177164:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334546944:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018926.177176:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018926.177182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018926/real 1734018926] req@ffff8800a78a1880 x1818247334546944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018926.177192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334546944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018926.177199:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018926.177201:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018926.177205:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334546944:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018926.177210:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334547072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018926.177224:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018926.177231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018926/real 1734018926] req@ffff8800a78a2680 x1818247334547072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018926.177241:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334547072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018926.177248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018926.177251:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018926.177254:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334547072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018926.177259:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334547200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018926.177271:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018926.177277:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018926/real 1734018926] req@ffff8800a78a0a80 x1818247334547200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018926.177286:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334547200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018926.177293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018926.177295:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018926.177298:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334547200:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018926.194741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018927.196720:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018928.198715:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018929.200686:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018930.202787:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018931.184600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018931.184610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018931.184615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018931.184626:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3197 00000100:00080000:2.0:1734018931.184634:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018931.184642:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018931.184680:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334547328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018931.184765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018931.184769:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018931.184771:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018931.184776:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3197 00000100:00080000:2.0:1734018931.184779:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018931.184784:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018931.184794:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018931.184807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334547456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018931.184817:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018931.184820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018931.184822:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018931.184826:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3197 00000100:00080000:2.0:1734018931.184828:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018931.184831:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018931.184837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018931.184846:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334547584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018931.184855:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018931.184858:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018931.184860:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018931.184863:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3197 00000100:00080000:2.0:1734018931.184866:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018931.184870:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018931.184874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018931.184876:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334547328:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018931.184882:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334547712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018931.184890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018931.184893:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018931.184895:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018931.184898:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3197 00000100:00080000:2.0:1734018931.184901:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018931.184904:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018931.184908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018931.184916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334547840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018931.184924:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018931.184927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018931.184929:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018931.184933:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3197 00000100:00080000:2.0:1734018931.184936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018931.184940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734018931.184942:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734018931.184944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018931.184951:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334547968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018931.184959:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018931.184962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018931.184963:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018931.184964:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018931/real 1734018931] req@ffff8800a78a0a80 x1818247334547328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018931.184967:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3197 00000100:00080000:2.0:1734018931.184969:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018931.184973:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018931.184976:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018931.184984:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334548096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018931.184992:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334547328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018931.185002:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018931.185005:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018931.185010:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334547328:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018931.185017:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334547456:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018931.185034:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018931.185041:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018931/real 1734018931] req@ffff8800a78a2680 x1818247334547456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018931.185052:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334547456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018931.185060:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018931.185062:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018931.185065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334547456:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018931.185071:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334547584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018931.185085:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018931.185091:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018931/real 1734018931] req@ffff8800a78a1880 x1818247334547584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018931.185101:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334547584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018931.185108:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018931.185111:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018931.185114:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334547584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018931.185139:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334547712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018931.185152:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018931.185158:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018931/real 1734018931] req@ffff8800a78a3b80 x1818247334547712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018931.185169:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334547712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018931.185176:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018931.185178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018931.185181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334547712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018931.185186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334547840:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018931.185198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018931.185204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018931/real 1734018931] req@ffff8800a78a0700 x1818247334547840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018931.185214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334547840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018931.185221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018931.185223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018931.185226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334547840:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018931.185231:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334547968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018931.185246:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018931.185252:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018931/real 1734018931] req@ffff8800a78a3480 x1818247334547968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018931.185263:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334547968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018931.185270:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018931.185272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018931.185275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334547968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018931.185280:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334548096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018931.185292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018931.185298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018931/real 1734018931] req@ffff8800a78a1f80 x1818247334548096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018931.185307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334548096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018931.185314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018931.185316:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018931.185319:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334548096:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018931.204779:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018932.206751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018933.208736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018934.210801:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018935.212741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018936.192646:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018936.192658:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018936.192663:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018936.192673:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3202 00000100:00080000:2.0:1734018936.192682:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018936.192689:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018936.192726:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334548224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018936.192768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018936.192772:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018936.192774:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018936.192779:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3202 00000100:00080000:2.0:1734018936.192782:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018936.192786:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018936.192795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018936.192807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334548352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018936.192816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018936.192819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018936.192821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018936.192825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3202 00000100:00080000:2.0:1734018936.192828:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018936.192831:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018936.192835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018936.192844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334548480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018936.192852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018936.192855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018936.192857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018936.192860:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3202 00000100:00080000:2.0:1734018936.192863:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018936.192867:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018936.192871:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018936.192879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334548608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018936.192884:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334548224:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018936.192888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018936.192891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018936.192892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018936.192896:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3202 00000100:00080000:2.0:1734018936.192898:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018936.192901:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018936.192905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018936.192912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334548736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018936.192920:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018936.192923:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018936.192925:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018936.192928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3202 00000100:00080000:2.0:1734018936.192931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018936.192934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018936.192938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018936.192946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334548864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018936.192948:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018936.192954:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018936.192957:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018936.192959:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018936.192962:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3202 00000100:00080000:2.0:1734018936.192965:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018936.192969:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018936/real 1734018936] req@ffff8800a78a1f80 x1818247334548224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018936.192969:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018936.192972:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018936.192979:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334548992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018936.192998:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334548224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018936.193008:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018936.193011:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018936.193016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334548224:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018936.193024:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334548352:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018936.193041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018936.193048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018936/real 1734018936] req@ffff8800a78a3480 x1818247334548352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018936.193059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334548352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018936.193067:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018936.193069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018936.193073:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334548352:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018936.193078:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334548480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018936.193092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018936.193098:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018936/real 1734018936] req@ffff8800a78a0700 x1818247334548480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018936.193108:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334548480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018936.193115:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018936.193118:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018936.193121:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334548480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018936.193146:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334548608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018936.193159:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018936.193165:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018936/real 1734018936] req@ffff8800a78a3b80 x1818247334548608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018936.193176:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334548608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018936.193183:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018936.193185:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018936.193188:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334548608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018936.193193:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334548736:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018936.193205:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018936.193211:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018936/real 1734018936] req@ffff8800a78a1880 x1818247334548736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018936.193221:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334548736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018936.193228:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018936.193230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018936.193233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334548736:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018936.193238:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334548864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018936.193253:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018936.193259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018936/real 1734018936] req@ffff8800a78a2680 x1818247334548864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018936.193269:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334548864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018936.193277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018936.193279:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018936.193282:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334548864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018936.193287:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334548992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018936.193299:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018936.193305:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018936/real 1734018936] req@ffff8800a78a0a80 x1818247334548992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018936.193314:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334548992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018936.193321:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018936.193323:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018936.193326:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334548992:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018936.214796:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018937.216772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018938.218805:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018939.220798:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018940.222811:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018941.200615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018941.200626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018941.200630:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018941.200641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3207 00000100:00080000:2.0:1734018941.200649:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018941.200657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018941.200695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334549120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018941.200721:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018941.200724:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018941.200725:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018941.200729:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3207 00000100:00080000:2.0:1734018941.200731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018941.200733:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018941.200741:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018941.200750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334549248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018941.200757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018941.200759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018941.200760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018941.200763:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3207 00000100:00080000:2.0:1734018941.200764:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018941.200767:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018941.200770:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018941.200776:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334549376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018941.200781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018941.200783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018941.200784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018941.200786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3207 00000100:00080000:2.0:1734018941.200788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018941.200806:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018941.200811:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018941.200821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334549504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018941.200831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018941.200834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018941.200836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018941.200839:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3207 00000100:00080000:2.0:1734018941.200843:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018941.200847:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018941.200851:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018941.200859:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334549632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018941.200869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018941.200872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018941.200873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018941.200877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3207 00000100:00080000:2.0:1734018941.200880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018941.200884:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018941.200885:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334549120:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734018941.200888:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018941.200896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334549760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018941.200904:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018941.200907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018941.200909:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018941.200913:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3207 00000100:00080000:2.0:1734018941.200915:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018941.200919:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018941.200923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018941.200931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334549888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018941.200959:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018941.200984:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018941/real 1734018941] req@ffff8800a78a0a80 x1818247334549120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018941.201012:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334549120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018941.201023:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018941.201027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018941.201032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334549120:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018941.201040:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334549248:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018941.201057:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018941.201064:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018941/real 1734018941] req@ffff8800a78a2680 x1818247334549248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018941.201075:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334549248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018941.201083:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018941.201085:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018941.201089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334549248:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018941.201095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334549376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018941.201109:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018941.201115:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018941/real 1734018941] req@ffff8800a78a1880 x1818247334549376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018941.201125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334549376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018941.201133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018941.201136:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018941.201139:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334549376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018941.201146:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334549504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018941.201159:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018941.201165:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018941/real 1734018941] req@ffff8800a78a3b80 x1818247334549504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018941.201175:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334549504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018941.201183:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018941.201185:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018941.201188:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334549504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018941.201215:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334549632:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018941.201229:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018941.201235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018941/real 1734018941] req@ffff8800a78a0700 x1818247334549632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018941.201246:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334549632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018941.201254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018941.201256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018941.201259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334549632:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018941.201265:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334549760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018941.201280:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018941.201287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018941/real 1734018941] req@ffff8800a78a3480 x1818247334549760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018941.201297:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334549760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018941.201304:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018941.201306:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018941.201309:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334549760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018941.201314:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334549888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018941.201326:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018941.201332:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018941/real 1734018941] req@ffff8800a78a1f80 x1818247334549888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018941.201342:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334549888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018941.201349:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018941.201351:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018941.201354:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334549888:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018941.224788:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018942.226752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018943.228744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018944.230766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018945.232759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018946.208591:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018946.208603:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018946.208607:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018946.208618:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3212 00000100:00080000:2.0:1734018946.208627:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018946.208634:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018946.208672:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334550016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018946.208705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018946.208709:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018946.208711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018946.208716:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3212 00000100:00080000:2.0:1734018946.208719:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018946.208722:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018946.208732:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018946.208743:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334550144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018946.208753:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018946.208756:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018946.208758:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018946.208762:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3212 00000100:00080000:2.0:1734018946.208764:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018946.208768:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018946.208773:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018946.208782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334550272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018946.208791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018946.208794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018946.208796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018946.208799:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3212 00000100:00080000:2.0:1734018946.208802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018946.208805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018946.208809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018946.208814:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334550016:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018946.208817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334550400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018946.208826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018946.208829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018946.208830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018946.208833:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3212 00000100:00080000:2.0:1734018946.208836:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018946.208839:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018946.208843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018946.208850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334550528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018946.208859:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018946.208861:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018946.208864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018946.208868:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3212 00000100:00080000:2.0:1734018946.208871:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018946.208874:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018946.208878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018946.208880:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018946.208886:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334550656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018946.208893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018946.208896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018946.208898:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018946.208901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3212 00000100:00100000:1.0:1734018946.208902:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018946/real 1734018946] req@ffff8800a78a1f80 x1818247334550016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018946.208904:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018946.208907:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018946.208911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018946.208918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334550784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018946.208930:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334550016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018946.208940:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018946.208943:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018946.208948:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334550016:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018946.208955:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334550144:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018946.208972:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018946.208979:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018946/real 1734018946] req@ffff8800a78a3480 x1818247334550144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018946.208990:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334550144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018946.208997:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018946.208999:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018946.209003:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334550144:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018946.209009:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334550272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018946.209022:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018946.209028:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018946/real 1734018946] req@ffff8800a78a0700 x1818247334550272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018946.209038:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334550272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018946.209045:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018946.209049:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018946.209052:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334550272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018946.209076:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334550400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018946.209089:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018946.209096:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018946/real 1734018946] req@ffff8800a78a3b80 x1818247334550400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018946.209106:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334550400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018946.209113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018946.209115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018946.209118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334550400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018946.209124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334550528:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018946.209136:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018946.209142:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018946/real 1734018946] req@ffff8800a78a1880 x1818247334550528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018946.209151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334550528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018946.209158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018946.209160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018946.209163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334550528:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018946.209169:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334550656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018946.209184:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018946.209190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018946/real 1734018946] req@ffff8800a78a2680 x1818247334550656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018946.209201:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334550656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018946.209208:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018946.209210:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018946.209213:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334550656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018946.209218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334550784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018946.209230:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018946.209236:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018946/real 1734018946] req@ffff8800a78a0a80 x1818247334550784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018946.209246:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334550784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018946.209253:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018946.209255:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018946.209258:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334550784:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018946.234725:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018947.236737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018948.238740:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018949.240754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018950.242742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018951.216692:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018951.216703:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018951.216708:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018951.216718:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3217 00000100:00080000:0.0:1734018951.216727:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018951.216734:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018951.216769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334550912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018951.216809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018951.216815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018951.216818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018951.216824:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3217 00000100:00080000:0.0:1734018951.216829:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018951.216834:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018951.216847:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018951.216865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334551040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018951.216877:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334550912:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018951.216881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018951.216886:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018951.216889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018951.216894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3217 00000100:00080000:0.0:1734018951.216898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018951.216903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018951.216910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018951.216922:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334551168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018951.216934:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018951.216939:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018951.216941:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018951.216946:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3217 00000800:00000100:1.0:1734018951.216948:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018951.216950:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018951.216955:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018951.216960:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018951.216972:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334551296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018951.216972:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018951/real 1734018951] req@ffff8800a934d180 x1818247334550912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018951.216985:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018951.216991:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018951.216993:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018951.216998:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3217 00000100:00080000:0.0:1734018951.217002:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018951.217007:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734018951.217007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334550912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018951.217014:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734018951.217021:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018951.217025:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018951.217027:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334551424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018951.217032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334550912:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734018951.217041:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018951.217045:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018951.217047:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018951.217052:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3217 00000100:00080000:0.0:1734018951.217056:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018951.217058:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334551040:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734018951.217061:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018951.217068:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018951.217080:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334551552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018951.217083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734018951.217095:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018951.217095:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018951/real 1734018951] req@ffff8800a934c000 x1818247334551040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018951.217100:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018951.217102:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018951.217108:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3217 00000100:00080000:0.0:1734018951.217112:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018951.217112:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334551040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018951.217117:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018951.217124:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018951.217124:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018951.217128:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018951.217134:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334551040:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018951.217135:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334551680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018951.217144:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334551168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018951.217167:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018951.217178:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018951/real 1734018951] req@ffff8800a934dc00 x1818247334551168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018951.217196:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334551168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018951.217208:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018951.217212:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018951.217218:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334551168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018951.217227:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334551296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018951.217251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018951.217260:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018951/real 1734018951] req@ffff8800a934e300 x1818247334551296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018951.217271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334551296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018951.217278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018951.217281:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018951.217284:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334551296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018951.217289:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334551424:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018951.217303:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018951.217309:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018951/real 1734018951] req@ffff8800a934f100 x1818247334551424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018951.217319:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334551424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018951.217327:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018951.217329:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018951.217333:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334551424:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018951.217407:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334551552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018951.217430:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018951.217438:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018951/real 1734018951] req@ffff8800a934fb80 x1818247334551552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018951.217449:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334551552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018951.217457:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018951.217459:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018951.217462:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334551552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018951.217510:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334551680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018951.217526:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018951.217533:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018951/real 1734018951] req@ffff8800a934d500 x1818247334551680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018951.217543:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334551680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018951.217551:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018951.217553:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018951.217556:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334551680:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018951.244783:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018952.246818:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018953.248818:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018954.250776:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018955.252788:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018956.224618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018956.224630:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018956.224634:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018956.224644:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3222 00000100:00080000:0.0:1734018956.224653:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018956.224662:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018956.224696:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334551808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018956.224730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018956.224734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018956.224736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018956.224741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3222 00000100:00080000:0.0:1734018956.224744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018956.224747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018956.224757:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018956.224767:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334551936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018956.224777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018956.224780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018956.224782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018956.224785:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3222 00000100:00080000:0.0:1734018956.224788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018956.224791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018956.224795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734018956.224805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334552064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018956.224813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018956.224817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018956.224819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018956.224822:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3222 00000100:00080000:0.0:1734018956.224825:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018956.224828:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018956.224832:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734018956.224838:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334551808:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018956.224840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334552192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018956.224849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018956.224852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018956.224853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018956.224856:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3222 00000100:00080000:0.0:1734018956.224859:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018956.224862:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018956.224866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018956.224874:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334552320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018956.224882:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018956.224885:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018956.224886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018956.224889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3222 00000100:00080000:0.0:1734018956.224892:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018956.224895:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018956.224899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018956.224902:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018956.224907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334552448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018956.224915:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018956.224918:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018956.224920:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018956.224923:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3222 00000100:00100000:1.0:1734018956.224923:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018956/real 1734018956] req@ffff8800a934e680 x1818247334551808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018956.224926:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018956.224929:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018956.224933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018956.224940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334552576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018956.224950:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334551808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018956.224960:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018956.224964:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018956.224969:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334551808:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018956.224976:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334551936:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018956.224993:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018956.225000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018956/real 1734018956] req@ffff8800a934ce00 x1818247334551936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018956.225011:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334551936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018956.225018:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018956.225021:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018956.225024:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334551936:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018956.225030:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334552064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018956.225043:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018956.225049:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018956/real 1734018956] req@ffff8800a934d500 x1818247334552064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018956.225059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334552064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018956.225066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018956.225069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018956.225072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334552064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018956.225100:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334552192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018956.225113:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018956.225120:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018956/real 1734018956] req@ffff8800a934fb80 x1818247334552192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018956.225130:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334552192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018956.225137:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018956.225139:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018956.225142:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334552192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018956.225148:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334552320:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018956.225159:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018956.225165:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018956/real 1734018956] req@ffff8800a934f100 x1818247334552320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018956.225175:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334552320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018956.225182:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018956.225184:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018956.225187:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334552320:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018956.225193:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334552448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018956.225208:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018956.225214:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018956/real 1734018956] req@ffff8800a934e300 x1818247334552448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018956.225224:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334552448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018956.225231:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018956.225233:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018956.225236:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334552448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018956.225241:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334552576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018956.225253:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018956.225259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018956/real 1734018956] req@ffff8800a934dc00 x1818247334552576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018956.225319:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334552576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018956.225327:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018956.225329:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018956.225332:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334552576:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018956.254777:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018957.256781:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018958.258816:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018959.260776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018960.262802:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018961.232696:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018961.232707:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018961.232711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018961.232722:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3227 00000100:00080000:2.0:1734018961.232730:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018961.232738:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018961.232775:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334552704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018961.232821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018961.232825:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018961.232827:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018961.232831:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3227 00000100:00080000:2.0:1734018961.232834:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018961.232838:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018961.232848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018961.232860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334552832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018961.232869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018961.232872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018961.232874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018961.232878:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3227 00000100:00080000:2.0:1734018961.232880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018961.232884:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018961.232889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018961.232897:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334552960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018961.232906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018961.232909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018961.232910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018961.232914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3227 00000100:00080000:2.0:1734018961.232916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018961.232920:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018961.232924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018961.232932:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334553088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018961.232938:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334552704:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018961.232941:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018961.232944:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018961.232945:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018961.232949:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3227 00000100:00080000:2.0:1734018961.232951:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018961.232954:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018961.232958:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018961.232965:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334553216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018961.232973:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018961.232976:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018961.232977:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018961.232981:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3227 00000100:00080000:2.0:1734018961.232983:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018961.232987:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018961.232991:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018961.232998:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334553344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018961.233004:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018961.233006:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018961.233010:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018961.233011:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018961.233015:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3227 00000100:00100000:1.0:1734018961.233026:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018961/real 1734018961] req@ffff8800a78a0a80 x1818247334552704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018961.233027:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018961.233031:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018961.233035:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018961.233043:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334553472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018961.233053:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334552704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018961.233063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018961.233066:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018961.233071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334552704:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734018961.233078:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334552832:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018961.233094:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018961.233101:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018961/real 1734018961] req@ffff8800a78a2680 x1818247334552832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018961.233112:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334552832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018961.233120:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018961.233122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018961.233126:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334552832:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018961.233132:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334552960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018961.233145:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018961.233151:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018961/real 1734018961] req@ffff8800a78a1880 x1818247334552960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018961.233161:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334552960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018961.233168:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018961.233172:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018961.233218:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334552960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018961.233245:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334553088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018961.233262:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018961.233270:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018961/real 1734018961] req@ffff8800a78a3b80 x1818247334553088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018961.233280:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334553088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018961.233288:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018961.233290:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018961.233293:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334553088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018961.233299:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334553216:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734018961.233311:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018961.233317:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018961/real 1734018961] req@ffff8800a78a0700 x1818247334553216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018961.233327:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334553216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018961.233335:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018961.233337:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018961.233340:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334553216:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018961.233345:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334553344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018961.233360:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018961.233366:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018961/real 1734018961] req@ffff8800a78a3480 x1818247334553344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018961.233377:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334553344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018961.233384:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018961.233386:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018961.233389:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334553344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018961.233394:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334553472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734018961.233406:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734018961.233412:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018961/real 1734018961] req@ffff8800a78a1f80 x1818247334553472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018961.233422:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334553472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018961.233429:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018961.233431:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734018961.233434:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334553472:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018961.264765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018962.266796:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018963.268771:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018964.270712:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018965.272786:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018966.240598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018966.240609:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018966.240614:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018966.240625:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3232 00000100:00080000:2.0:1734018966.240633:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018966.240641:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018966.240679:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334553600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018966.240712:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018966.240716:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018966.240718:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018966.240722:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3232 00000100:00080000:2.0:1734018966.240725:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018966.240729:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018966.240738:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018966.240749:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334553728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018966.240760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018966.240764:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018966.240765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018966.240769:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3232 00000100:00080000:2.0:1734018966.240771:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018966.240775:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018966.240779:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018966.240788:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334553856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018966.240797:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018966.240800:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018966.240801:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018966.240805:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3232 00000100:00080000:2.0:1734018966.240807:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018966.240810:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018966.240814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018966.240822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334553984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018966.240826:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334553600:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018966.240831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018966.240834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018966.240836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018966.240839:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3232 00000100:00080000:2.0:1734018966.240842:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018966.240846:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018966.240850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018966.240857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334554112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018966.240866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018966.240868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018966.240870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018966.240873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3232 00000100:00080000:2.0:1734018966.240876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018966.240879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018966.240883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018966.240890:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334554240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018966.240898:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018966.240901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018966.240903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018966.240906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3232 00000100:00080000:2.0:1734018966.240909:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018966.240912:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018966.240916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018966.240919:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334553728:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018966.240923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334554368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018966.240941:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334553856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018966.240965:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334553984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018966.240980:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334554112:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734018966.240994:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334554240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734018966.241009:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334554368:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018966.274744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018967.276804:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018968.278772:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018969.246850:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734018969.246915:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018969.246942:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018969.246988:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018969.246998:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018969.247006:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018969.247014:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018969.247020:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734018969.247027:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:0.0:1734018969.247130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018966/real 1734018969] req@ffff8800a78a3b80 x1818247334553984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018969.247162:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334553984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018969.247174:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018969.247178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018969.247184:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334553984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018969.247194:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018966/real 1734018969] req@ffff8800a78a1880 x1818247334554112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018969.247214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334554112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018969.247226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018969.247230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018969.247237:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334554112:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018969.247247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018966/real 1734018969] req@ffff8800a78a2680 x1818247334554240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018969.247264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334554240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018969.247277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018969.247281:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018969.247285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334554240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018969.247295:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018966/real 1734018969] req@ffff8800a78a0a80 x1818247334554368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018969.247313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334554368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018969.247326:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018969.247329:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018969.247334:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334554368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018969.247344:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018966/real 1734018969] req@ffff8800a78a1f80 x1818247334553600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018969.247363:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334553600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018969.247374:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018969.247378:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018969.247383:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334553600:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018969.247394:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018966/real 1734018969] req@ffff8800a78a3480 x1818247334553728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018969.247413:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334553728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018969.247425:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018969.247429:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018969.247447:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334553728:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018969.247458:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018966/real 1734018969] req@ffff8800a78a0700 x1818247334553856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018969.247549:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334553856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018969.247562:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018969.247567:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018969.247572:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334553856:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734018969.280615:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018970.282812:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018971.248626:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018971.248639:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018971.248645:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018971.248660:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3237 00000100:00080000:2.0:1734018971.248671:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018971.248681:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018971.248732:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334554496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018971.248768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018971.248773:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018971.248775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018971.248780:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3237 00000100:00080000:2.0:1734018971.248783:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018971.248787:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018971.248798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018971.248810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334554624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018971.248820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018971.248823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018971.248825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018971.248830:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3237 00000100:00080000:2.0:1734018971.248833:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018971.248839:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018971.248844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018971.248854:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334554752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018971.248864:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018971.248867:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018971.248869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018971.248872:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3237 00000100:00080000:2.0:1734018971.248875:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018971.248879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734018971.248880:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334554496:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734018971.248883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018971.248891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334554880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018971.248900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018971.248903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018971.248905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018971.248908:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3237 00000100:00080000:2.0:1734018971.248911:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018971.248914:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018971.248919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018971.248927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334555008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018971.248935:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018971.248939:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018971.248941:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018971.248945:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3237 00000100:00080000:2.0:1734018971.248957:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018971.248961:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:0.0:1734018971.248965:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734018971.248966:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018971.248975:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334555136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018971.248984:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018971.248988:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018971.248989:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018971.248993:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018971/real 1734018971] req@ffff8800a78a0700 x1818247334554496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018971.248993:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3237 00000100:00080000:2.0:1734018971.248996:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018971.248999:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018971.249003:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018971.249040:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334555264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018971.249046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334554496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018971.249058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018971.249062:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018971.249067:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334554496:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018971.249075:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334554624:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018971.249096:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018971.249104:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018971/real 1734018971] req@ffff8800a78a3480 x1818247334554624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018971.249116:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334554624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018971.249124:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018971.249126:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018971.249130:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334554624:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018971.249156:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334554752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018971.249171:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018971.249177:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018971/real 1734018971] req@ffff8800a78a1f80 x1818247334554752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018971.249189:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334554752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018971.249196:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018971.249198:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018971.249201:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334554752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018971.249207:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334554880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018971.249224:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018971.249230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018971/real 1734018971] req@ffff8800a78a0a80 x1818247334554880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018971.249241:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334554880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018971.249248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018971.249250:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018971.249253:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334554880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018971.249259:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334555008:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018971.249272:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018971.249278:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018971/real 1734018971] req@ffff8800a78a2680 x1818247334555008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018971.249288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334555008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018971.249295:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018971.249297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018971.249300:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334555008:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018971.249305:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334555136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018971.249318:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018971.249324:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018971/real 1734018971] req@ffff8800a78a1880 x1818247334555136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018971.249341:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334555136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018971.249349:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018971.249351:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018971.249354:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334555136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018971.249359:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334555264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018971.249372:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018971.249378:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018971/real 1734018971] req@ffff8800a78a3b80 x1818247334555264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018971.249388:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334555264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018971.249395:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018971.249397:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018971.249400:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334555264:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734018971.284764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018972.286774:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018973.288751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018974.290780:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018975.292760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018976.256582:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018976.256593:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018976.256598:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018976.256608:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3242 00000100:00080000:2.0:1734018976.256617:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018976.256626:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018976.256663:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334555392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018976.256695:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018976.256700:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018976.256702:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018976.256706:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3242 00000100:00080000:2.0:1734018976.256709:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018976.256713:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018976.256722:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018976.256734:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334555520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018976.256744:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018976.256747:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018976.256749:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018976.256753:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3242 00000100:00080000:2.0:1734018976.256755:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018976.256759:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018976.256763:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018976.256771:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334555648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018976.256780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018976.256783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018976.256786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018976.256789:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3242 00000100:00080000:2.0:1734018976.256792:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018976.256795:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018976.256799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018976.256806:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334555392:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018976.256808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334555776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018976.256816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018976.256819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018976.256821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018976.256824:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3242 00000100:00080000:2.0:1734018976.256826:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018976.256830:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018976.256833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018976.256841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334555904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018976.256848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018976.256851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018976.256853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018976.256856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3242 00000100:00080000:2.0:1734018976.256868:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018976.256871:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:0.0:1734018976.256872:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734018976.256876:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018976.256883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334556032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018976.256892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734018976.256894:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018976/real 1734018976] req@ffff8800a78a3b80 x1818247334555392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018976.256896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018976.256897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018976.256901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3242 00000100:00080000:2.0:1734018976.256904:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018976.256907:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018976.256911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018976.256914:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334555392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018976.256918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334556160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018976.256953:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018976.256958:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018976.256963:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334555392:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018976.256971:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334555520:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018976.256992:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018976.257000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018976/real 1734018976] req@ffff8800a78a1880 x1818247334555520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018976.257012:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334555520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018976.257020:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018976.257023:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018976.257026:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334555520:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018976.257032:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334555648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018976.257045:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018976.257051:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018976/real 1734018976] req@ffff8800a78a2680 x1818247334555648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018976.257061:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334555648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018976.257069:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018976.257071:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018976.257075:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334555648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018976.257099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334555776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018976.257115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018976.257122:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018976/real 1734018976] req@ffff8800a78a0a80 x1818247334555776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018976.257132:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334555776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018976.257140:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018976.257142:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018976.257145:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334555776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018976.257150:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334555904:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018976.257162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018976.257168:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018976/real 1734018976] req@ffff8800a78a1f80 x1818247334555904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018976.257178:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334555904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018976.257185:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018976.257187:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018976.257190:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334555904:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018976.257196:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334556032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018976.257208:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018976.257214:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018976/real 1734018976] req@ffff8800a78a3480 x1818247334556032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018976.257231:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334556032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018976.257238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018976.257240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018976.257243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334556032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018976.257249:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334556160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018976.257261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018976.257267:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018976/real 1734018976] req@ffff8800a78a0700 x1818247334556160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018976.257277:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334556160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018976.257284:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018976.257286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018976.257289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334556160:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018976.294797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018977.296740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018978.298758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018979.300730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018980.302774:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018981.264728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018981.264739:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018981.264743:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018981.264753:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3247 00000100:00080000:0.0:1734018981.264762:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018981.264770:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734018981.264807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334556288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018981.264877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018981.264882:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018981.264883:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018981.264888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3247 00000100:00080000:0.0:1734018981.264892:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018981.264896:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018981.264906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018981.264918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334556416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018981.264928:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018981.264931:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018981.264933:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018981.264937:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3247 00000100:00080000:0.0:1734018981.264939:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018981.264943:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018981.264947:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018981.264952:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334556288:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018981.264956:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334556544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018981.264965:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018981.264968:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018981.264970:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018981.264973:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3247 00000100:00080000:0.0:1734018981.264975:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018981.264979:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018981.264984:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018981.264992:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334556672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018981.265000:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018981.265003:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018981.265004:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018981.265008:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3247 00000100:00080000:0.0:1734018981.265010:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018981.265013:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018981.265018:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1734018981.265019:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018981.265025:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334556800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018981.265033:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018981.265036:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018981.265037:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018981.265041:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3247 00000100:00100000:2.0:1734018981.265042:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018981/real 1734018981] req@ffff8800a934c000 x1818247334556288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018981.265043:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018981.265046:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734018981.265050:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018981.265058:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334556928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018981.265062:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334556288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018981.265066:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018981.265069:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018981.265071:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018981.265073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018981.265075:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3247 00000100:00080000:2.0:1734018981.265076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734018981.265077:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018981.265081:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734018981.265081:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334556288:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734018981.265085:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018981.265089:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334556416:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018981.265092:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334557056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734018981.265106:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018981.265114:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018981/real 1734018981] req@ffff8800a934d180 x1818247334556416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018981.265125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334556416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018981.265133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018981.265135:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018981.265139:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334556416:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018981.265161:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334556544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018981.265175:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018981.265182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018981/real 1734018981] req@ffff8800a934c700 x1818247334556544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018981.265192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334556544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018981.265199:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018981.265202:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018981.265205:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334556544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018981.265211:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334556672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018981.265223:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018981.265229:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018981/real 1734018981] req@ffff8800a934ed80 x1818247334556672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018981.265239:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334556672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018981.265254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018981.265256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018981.265259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334556672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018981.265264:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334556800:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734018981.265278:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018981.265284:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018981/real 1734018981] req@ffff8800a934ca80 x1818247334556800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018981.265294:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334556800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018981.265301:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018981.265303:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018981.265306:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334556800:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734018981.265312:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334556928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018981.265324:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018981.265330:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018981/real 1734018981] req@ffff8800a934ea00 x1818247334556928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018981.265340:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334556928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018981.265347:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018981.265350:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018981.265352:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334556928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734018981.265358:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334557056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734018981.265371:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018981.265377:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018981/real 1734018981] req@ffff8800a934c380 x1818247334557056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018981.265387:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334557056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018981.265394:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018981.265396:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734018981.265399:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334557056:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734018981.304747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018982.306792:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018983.308729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018984.310837:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018985.312753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018986.272694:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018986.272708:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018986.272714:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018986.272727:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3252 00000100:00080000:2.0:1734018986.272740:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018986.272789:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018986.272843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334557184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018986.272883:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018986.272888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018986.272890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018986.272895:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3252 00000100:00080000:2.0:1734018986.272900:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018986.272904:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018986.272915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018986.272928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334557312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018986.272938:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018986.272942:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018986.272944:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018986.272948:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3252 00000100:00080000:2.0:1734018986.272950:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018986.272955:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018986.272959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018986.272968:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334557440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018986.272977:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018986.272981:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018986.272982:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018986.272986:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3252 00000100:00100000:0.0:1734018986.272989:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334557184:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734018986.272989:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018986.272992:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018986.272996:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018986.273004:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334557568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018986.273020:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018986.273023:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018986.273025:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018986.273028:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3252 00000100:00080000:2.0:1734018986.273031:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018986.273034:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018986.273038:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018986.273046:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334557696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018986.273054:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018986.273057:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018986.273058:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018986.273062:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3252 00000100:00080000:2.0:1734018986.273065:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018986.273068:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018986.273072:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734018986.273075:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018986.273080:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334557824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018986.273088:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018986.273091:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018986.273093:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018986.273096:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3252 00000100:00080000:2.0:1734018986.273100:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734018986.273102:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018986/real 1734018986] req@ffff8800a78a0700 x1818247334557184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018986.273104:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018986.273108:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018986.273117:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334557952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018986.273125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334557184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018986.273137:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018986.273141:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018986.273146:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334557184:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018986.273154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334557312:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018986.273172:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018986.273179:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018986/real 1734018986] req@ffff8800a78a3480 x1818247334557312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018986.273190:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334557312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018986.273200:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018986.273202:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018986.273206:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334557312:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018986.273233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334557440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018986.273248:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018986.273254:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018986/real 1734018986] req@ffff8800a78a1f80 x1818247334557440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018986.273265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334557440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018986.273273:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018986.273275:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018986.273278:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334557440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018986.273284:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334557568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018986.273296:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018986.273302:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018986/real 1734018986] req@ffff8800a78a0a80 x1818247334557568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018986.273312:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334557568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018986.273319:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018986.273322:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018986.273325:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334557568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018986.273331:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334557696:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018986.273344:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018986.273350:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018986/real 1734018986] req@ffff8800a78a2680 x1818247334557696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018986.273361:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334557696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018986.273368:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018986.273370:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018986.273373:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334557696:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018986.273378:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334557824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018986.273391:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018986.273397:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018986/real 1734018986] req@ffff8800a78a1880 x1818247334557824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018986.273407:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334557824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018986.273414:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018986.273416:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018986.273419:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334557824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018986.273425:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334557952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018986.273438:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018986.273444:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018986/real 1734018986] req@ffff8800a78a3b80 x1818247334557952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018986.273454:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334557952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018986.273461:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018986.273494:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018986.273498:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334557952:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018986.314731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018987.316698:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018988.318828:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018989.320769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018990.322777:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018991.280585:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018991.280598:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018991.280604:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018991.280618:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3257 00000100:00080000:2.0:1734018991.280629:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018991.280639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018991.280721:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334558080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018991.280759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018991.280763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018991.280765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018991.280770:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3257 00000100:00080000:2.0:1734018991.280774:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018991.280778:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018991.280790:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018991.280803:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334558208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018991.280820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018991.280824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018991.280825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018991.280830:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3257 00000100:00080000:2.0:1734018991.280832:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018991.280836:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018991.280841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018991.280851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334558336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018991.280860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018991.280864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018991.280865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018991.280869:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3257 00000100:00080000:2.0:1734018991.280872:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018991.280875:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734018991.280880:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334558080:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734018991.280880:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734018991.280889:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334558464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018991.280898:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018991.280901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018991.280903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018991.280906:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3257 00000100:00080000:2.0:1734018991.280910:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018991.280913:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018991.280918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018991.280926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334558592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018991.280934:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018991.280937:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018991.280939:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018991.280942:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3257 00000100:00080000:2.0:1734018991.280946:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018991.280949:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018991.280953:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734018991.280954:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734018991.280961:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334558720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018991.280970:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018991.280973:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018991.280975:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018991.280979:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3257 00000100:00100000:0.0:1734018991.280980:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018991/real 1734018991] req@ffff8800a78a3b80 x1818247334558080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018991.280982:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018991.280985:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018991.280989:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018991.280997:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334558848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018991.281000:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334558080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018991.281011:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018991.281015:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018991.281020:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334558080:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018991.281029:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334558208:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018991.281048:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018991.281055:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018991/real 1734018991] req@ffff8800a78a1880 x1818247334558208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018991.281066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334558208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018991.281074:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018991.281077:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018991.281081:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334558208:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018991.281106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334558336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018991.281120:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018991.281127:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018991/real 1734018991] req@ffff8800a78a2680 x1818247334558336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018991.281138:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334558336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018991.281145:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018991.281148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018991.281151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334558336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018991.281158:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334558464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018991.281170:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018991.281176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018991/real 1734018991] req@ffff8800a78a0a80 x1818247334558464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018991.281186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334558464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018991.281194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018991.281196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018991.281199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334558464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018991.281206:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334558592:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018991.281223:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018991.281230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018991/real 1734018991] req@ffff8800a78a1f80 x1818247334558592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018991.281240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334558592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018991.281248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018991.281250:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018991.281253:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334558592:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018991.281259:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334558720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018991.281272:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018991.281278:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018991/real 1734018991] req@ffff8800a78a3480 x1818247334558720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018991.281288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334558720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018991.281295:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018991.281297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018991.281300:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334558720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018991.281306:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334558848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018991.281321:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018991.281327:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018991/real 1734018991] req@ffff8800a78a0700 x1818247334558848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018991.281343:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334558848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018991.281351:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018991.281353:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018991.281356:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334558848:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734018991.324750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018992.326773:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018993.328788:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018994.330823:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018995.332704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018996.288622:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018996.288632:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018996.288637:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018996.288647:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3262 00000100:00080000:2.0:1734018996.288656:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018996.288663:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734018996.288700:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334558976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018996.288731:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018996.288735:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018996.288737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018996.288742:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3262 00000100:00080000:2.0:1734018996.288745:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018996.288749:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018996.288758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018996.288769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334559104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018996.288778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018996.288781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018996.288783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018996.288787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3262 00000100:00080000:2.0:1734018996.288791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018996.288794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018996.288799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734018996.288807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334559232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018996.288816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018996.288819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018996.288820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018996.288823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3262 00000100:00080000:2.0:1734018996.288826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018996.288829:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018996.288833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734018996.288835:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334558976:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734018996.288841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334559360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018996.288850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018996.288853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018996.288854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018996.288857:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3262 00000100:00080000:2.0:1734018996.288860:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018996.288863:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018996.288867:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018996.288874:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334559488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018996.288889:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018996.288892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018996.288894:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018996.288897:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3262 00000800:00000100:0.0:1734018996.288900:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734018996.288900:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018996.288903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018996.288907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018996.288914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334559616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018996.288922:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734018996.288923:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018996/real 1734018996] req@ffff8800a78a0700 x1818247334558976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018996.288925:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018996.288927:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018996.288930:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3262 00000100:00080000:2.0:1734018996.288933:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018996.288936:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734018996.288939:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018996.288942:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334558976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018996.288947:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334559744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018996.288953:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018996.288956:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018996.288961:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334558976:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734018996.288969:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334559104:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018996.288985:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018996.288992:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018996/real 1734018996] req@ffff8800a78a3480 x1818247334559104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018996.289003:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334559104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018996.289011:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018996.289014:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018996.289018:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334559104:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018996.289024:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334559232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018996.289038:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018996.289044:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018996/real 1734018996] req@ffff8800a78a1f80 x1818247334559232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018996.289054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334559232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018996.289061:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018996.289064:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018996.289068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334559232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018996.289091:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334559360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018996.289104:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018996.289110:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018996/real 1734018996] req@ffff8800a78a0a80 x1818247334559360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018996.289121:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334559360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018996.289129:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018996.289131:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018996.289134:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334559360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018996.289139:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334559488:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734018996.289154:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018996.289160:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018996/real 1734018996] req@ffff8800a78a2680 x1818247334559488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018996.289171:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334559488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018996.289178:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018996.289180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018996.289183:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334559488:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734018996.289188:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334559616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018996.289200:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018996.289206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018996/real 1734018996] req@ffff8800a78a1880 x1818247334559616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018996.289215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334559616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018996.289222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018996.289224:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018996.289227:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334559616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734018996.289232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334559744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734018996.289245:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734018996.289251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018996/real 1734018996] req@ffff8800a78a3b80 x1818247334559744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018996.289267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334559744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018996.289275:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018996.289277:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734018996.289280:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334559744:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734018996.334766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018997.336808:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018998.338858:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018999.340765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019000.342789:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019001.296598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019001.296612:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019001.296617:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019001.296631:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3267 00000100:00080000:2.0:1734019001.296642:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019001.296651:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019001.296696:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334559872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019001.296736:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019001.296742:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019001.296745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019001.296751:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3267 00000100:00080000:2.0:1734019001.296757:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019001.296763:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019001.296775:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019001.296791:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334559872:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019001.296794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334560000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019001.296808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019001.296813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019001.296815:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019001.296821:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3267 00000100:00080000:2.0:1734019001.296825:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019001.296829:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019001.296835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019001.296848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334560128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019001.296861:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019001.296861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019001.296866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019001.296869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019001.296873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3267 00000100:00100000:0.0:1734019001.296886:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019001/real 1734019001] req@ffff8800a78a3b80 x1818247334559872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019001.296892:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019001.296897:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019001.296904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019001.296910:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334559872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019001.296918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334560256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019001.296925:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019001.296929:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019001.296932:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019001.296936:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334559872:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019001.296937:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019001.296939:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019001.296944:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3267 00000100:00080000:2.0:1734019001.296948:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019001.296953:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019001.296959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019001.296964:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334560000:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019001.296970:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334560384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019001.296984:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019001.296989:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1734019001.296992:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019001.296992:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019001.296997:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3267 00000100:00080000:2.0:1734019001.297001:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019001.297004:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019001/real 1734019001] req@ffff8800a78a1880 x1818247334560000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019001.297007:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019001.297014:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019001.297021:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334560000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019001.297026:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334560512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019001.297034:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019001.297038:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019001.297040:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019001.297044:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334560000:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019001.297046:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019001.297048:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019001.297054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334560128:192.168.201.116@tcp:8:kworker.0 00000100:00080000:2.0:1734019001.297054:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3267 00000100:00080000:2.0:1734019001.297058:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019001.297062:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019001.297068:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019001.297077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019001.297080:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334560640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019001.297088:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019001/real 1734019001] req@ffff8800a78a2680 x1818247334560128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019001.297105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334560128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019001.297117:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019001.297120:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019001.297125:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334560128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019001.297134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334560256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019001.297162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019001.297174:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019001/real 1734019001] req@ffff8800a78a0a80 x1818247334560256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019001.297186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334560256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019001.297194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019001.297197:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019001.297200:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334560256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019001.297218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334560384:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734019001.297238:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019001.297245:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019001/real 1734019001] req@ffff8800a78a1f80 x1818247334560384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019001.297255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334560384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019001.297262:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019001.297265:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019001.297268:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334560384:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019001.297273:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334560512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019001.297285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019001.297291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019001/real 1734019001] req@ffff8800a78a3480 x1818247334560512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019001.297301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334560512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019001.297308:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019001.297310:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019001.297313:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334560512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019001.297318:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334560640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019001.297330:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019001.297336:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019001/real 1734019001] req@ffff8800a78a0700 x1818247334560640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019001.297353:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334560640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019001.297361:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019001.297363:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019001.297366:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334560640:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019001.344712:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019002.346786:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019003.348738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019004.350798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019005.352739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019006.304625:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019006.304639:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019006.304645:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019006.304658:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3272 00000100:00080000:2.0:1734019006.304669:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019006.304679:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019006.304730:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334560768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019006.304770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019006.304778:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019006.304780:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019006.304785:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3272 00000100:00080000:2.0:1734019006.304789:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019006.304793:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019006.304804:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019006.304817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334560896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019006.304827:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019006.304831:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019006.304832:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019006.304836:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3272 00000100:00080000:2.0:1734019006.304839:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019006.304844:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019006.304848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019006.304859:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334561024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019006.304867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019006.304870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019006.304872:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019006.304875:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3272 00000100:00080000:2.0:1734019006.304878:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019006.304882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734019006.304885:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334560768:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019006.304888:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019006.304896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334561152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019006.304906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019006.304909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019006.304911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019006.304914:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3272 00000100:00080000:2.0:1734019006.304917:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019006.304920:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019006.304924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019006.304932:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334561280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019006.304940:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019006.304943:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019006.304944:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019006.304948:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3272 00000100:00080000:2.0:1734019006.304951:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019006.304955:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019006.304959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019006.304967:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334561408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019006.304973:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019006.304974:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019006.304978:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019006.304980:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019006.304984:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3272 00000100:00080000:2.0:1734019006.304987:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019006.304990:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019006.304994:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019006.305002:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019006/real 1734019006] req@ffff8800a78a0700 x1818247334560768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019006.305002:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334561536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019006.305025:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334560768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019006.305036:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019006.305040:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019006.305045:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334560768:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734019006.305053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334560896:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734019006.305071:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019006.305078:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019006/real 1734019006] req@ffff8800a78a3480 x1818247334560896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019006.305089:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334560896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019006.305097:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019006.305101:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019006.305105:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334560896:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019006.305111:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334561024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019006.305126:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019006.305132:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019006/real 1734019006] req@ffff8800a78a1f80 x1818247334561024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019006.305142:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334561024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019006.305150:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019006.305153:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019006.305156:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334561024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019006.305185:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334561152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019006.305198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019006.305205:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019006/real 1734019006] req@ffff8800a78a0a80 x1818247334561152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019006.305215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334561152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019006.305224:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019006.305226:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019006.305229:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334561152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019006.305234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334561280:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734019006.305250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019006.305257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019006/real 1734019006] req@ffff8800a78a2680 x1818247334561280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019006.305267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334561280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019006.305274:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019006.305276:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019006.305279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334561280:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019006.305284:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334561408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019006.305297:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019006.305303:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019006/real 1734019006] req@ffff8800a78a1880 x1818247334561408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019006.305313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334561408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019006.305320:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019006.305322:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019006.305325:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334561408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019006.305330:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334561536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019006.305343:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019006.305349:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019006/real 1734019006] req@ffff8800a78a3b80 x1818247334561536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019006.305367:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334561536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019006.305375:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019006.305377:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019006.305380:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334561536:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019006.354734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019007.356753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019008.358700:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019009.360716:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019010.362772:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019011.312701:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019011.312712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019011.312716:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019011.312727:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3277 00000100:00080000:0.0:1734019011.312735:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019011.312743:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019011.312779:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334561664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019011.312819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019011.312823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019011.312825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019011.312830:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3277 00000100:00080000:0.0:1734019011.312833:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019011.312836:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019011.312846:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019011.312857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334561792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019011.312864:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334561664:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734019011.312874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019011.312878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019011.312880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019011.312884:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3277 00000100:00080000:0.0:1734019011.312886:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019011.312890:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019011.312894:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019011.312904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334561920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019011.312913:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019011.312916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019011.312918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019011.312921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3277 00000100:00080000:0.0:1734019011.312924:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019011.312927:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734019011.312930:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734019011.312931:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019011.312939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334562048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019011.312948:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019011.312951:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019011.312952:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019011.312954:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019011/real 1734019011] req@ffff8800a934df80 x1818247334561664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019011.312956:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3277 00000100:00080000:0.0:1734019011.312959:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019011.312963:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019011.312967:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019011.312973:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334561664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019011.312975:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334562176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019011.312982:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019011.312984:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019011.312985:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019011.312987:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019011.312987:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734019011.312990:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3277 00000100:00100000:2.0:1734019011.312992:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334561664:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734019011.312993:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019011.312996:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019011.313000:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019011.313007:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334562304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019011.313014:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334561792:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734019011.313016:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019011.313019:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019011.313020:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019011.313024:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3277 00000100:00080000:0.0:1734019011.313026:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019011.313029:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:2.0:1734019011.313032:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:0.0:1734019011.313033:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019011.313040:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019011/real 1734019011] req@ffff8800a934d880 x1818247334561792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019011.313041:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334562432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019011.313052:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334561792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019011.313060:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019011.313063:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019011.313066:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334561792:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019011.313072:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334561920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019011.313086:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019011.313092:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019011/real 1734019011] req@ffff8800a934c380 x1818247334561920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019011.313102:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334561920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019011.313110:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019011.313112:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019011.313115:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334561920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019011.313121:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334562048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019011.313134:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019011.313140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019011/real 1734019011] req@ffff8800a934ea00 x1818247334562048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019011.313150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334562048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019011.313158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019011.313160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019011.313163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334562048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019011.313170:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334562176:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019011.313183:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019011.313190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019011/real 1734019011] req@ffff8800a934ca80 x1818247334562176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019011.313200:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334562176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019011.313207:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019011.313209:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019011.313212:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334562176:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019011.313227:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334562304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019011.313239:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019011.313245:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019011/real 1734019011] req@ffff8800a934ed80 x1818247334562304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019011.313255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334562304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019011.313262:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019011.313264:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019011.313267:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334562304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019011.313274:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334562432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019011.313286:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019011.313293:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019011/real 1734019011] req@ffff8800a934c700 x1818247334562432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019011.313304:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334562432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019011.313312:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019011.313314:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019011.313317:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334562432:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019011.364734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019012.366718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019013.368735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019014.370769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019015.372747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019016.320603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019016.320615:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019016.320620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019016.320634:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3282 00000100:00080000:0.0:1734019016.320645:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019016.320655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019016.320701:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334562560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019016.320737:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019016.320741:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019016.320743:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019016.320748:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3282 00000100:00080000:0.0:1734019016.320752:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019016.320756:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019016.320767:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019016.320779:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334562688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019016.320789:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019016.320792:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019016.320794:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019016.320799:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3282 00000100:00080000:0.0:1734019016.320802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019016.320805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019016.320812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019016.320821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334562816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019016.320831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019016.320834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019016.320836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019016.320839:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3282 00000100:00080000:0.0:1734019016.320842:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019016.320845:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734019016.320845:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334562560:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734019016.320850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019016.320859:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334562944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019016.320867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019016.320870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019016.320871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019016.320875:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3282 00000100:00080000:0.0:1734019016.320878:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019016.320881:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019016.320886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019016.320897:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334563072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019016.320904:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019016.320907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019016.320910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019016.320914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3282 00000100:00080000:0.0:1734019016.320917:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019016.320921:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019016.320925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019016.320928:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019016.320933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334563200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019016.320941:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019016.320944:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019016.320945:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019016.320949:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3282 00000100:00080000:0.0:1734019016.320951:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019016.320955:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734019016.320955:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019016/real 1734019016] req@ffff8800a934d180 x1818247334562560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019016.320959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019016.320967:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334563328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019016.320977:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334562560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019016.320989:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019016.320993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019016.320998:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334562560:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019016.321005:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334562688:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019016.321023:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019016.321030:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019016/real 1734019016] req@ffff8800a934c000 x1818247334562688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019016.321042:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334562688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019016.321052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019016.321055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019016.321058:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334562688:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019016.321083:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334562816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019016.321098:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019016.321104:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019016/real 1734019016] req@ffff8800a934dc00 x1818247334562816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019016.321115:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334562816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019016.321123:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019016.321126:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019016.321129:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334562816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019016.321134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334562944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019016.321146:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019016.321152:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019016/real 1734019016] req@ffff8800a934e300 x1818247334562944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019016.321167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334562944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019016.321179:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019016.321182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019016.321199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334562944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019016.321209:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334563072:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019016.321265:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019016.321281:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019016/real 1734019016] req@ffff8800a934f100 x1818247334563072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019016.321302:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334563072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019016.321316:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019016.321319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019016.321325:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334563072:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019016.321335:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334563200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019016.321363:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019016.321373:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019016/real 1734019016] req@ffff8800a934fb80 x1818247334563200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019016.321384:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334563200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019016.321393:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019016.321395:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019016.321398:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334563200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019016.321403:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334563328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019016.321417:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019016.321424:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019016/real 1734019016] req@ffff8800a934d500 x1818247334563328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019016.321435:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334563328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019016.321443:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019016.321445:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019016.321448:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334563328:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019016.374739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019017.376794:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019018.378807:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019019.380798:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019020.382726:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019021.328726:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019021.328738:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019021.328744:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019021.328757:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3287 00000100:00080000:2.0:1734019021.328769:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019021.328779:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019021.328832:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334563456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019021.328877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019021.328882:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019021.328883:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019021.328888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3287 00000100:00080000:2.0:1734019021.328892:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019021.328896:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019021.328907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019021.328920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334563584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019021.328934:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334563456:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019021.328934:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019021.328937:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019021.328939:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019021.328943:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3287 00000100:00080000:2.0:1734019021.328946:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019021.328950:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019021.328955:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019021.328964:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334563712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019021.328973:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019021.328977:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019021.328978:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019021.328981:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3287 00000100:00080000:2.0:1734019021.328984:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019021.328988:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019021.328993:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019021.329001:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334563840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019021.329007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019021.329010:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019021.329013:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019021.329015:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019021.329018:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3287 00000100:00080000:2.0:1734019021.329025:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019021.329029:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734019021.329030:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019021/real 1734019021] req@ffff8800a78a3b80 x1818247334563456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019021.329034:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019021.329042:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334563968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019021.329050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334563456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019021.329051:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019021.329054:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019021.329055:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019021.329059:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3287 00000100:00080000:0.0:1734019021.329062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019021.329062:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019021.329065:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:0.0:1734019021.329066:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000008:00100000:2.0:1734019021.329070:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019021.329071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334563456:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019021.329078:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334564096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019021.329086:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019021.329090:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019021.329091:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019021.329095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334563584:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019021.329095:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3287 00000100:00080000:2.0:1734019021.329097:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019021.329101:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019021.329105:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019021.329114:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019021.329114:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334564224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019021.329121:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019021/real 1734019021] req@ffff8800a78a1880 x1818247334563584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019021.329134:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334563584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019021.329142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019021.329145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019021.329149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334563584:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019021.329183:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334563712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019021.329200:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019021.329208:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019021/real 1734019021] req@ffff8800a78a2680 x1818247334563712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019021.329218:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334563712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019021.329226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019021.329229:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019021.329233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334563712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019021.329239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334563840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019021.329255:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019021.329262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019021/real 1734019021] req@ffff8800a78a0a80 x1818247334563840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019021.329272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334563840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019021.329280:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019021.329282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019021.329285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334563840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019021.329291:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334563968:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734019021.329304:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019021.329310:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019021/real 1734019021] req@ffff8800a78a1f80 x1818247334563968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019021.329321:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334563968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019021.329328:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019021.329331:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019021.329334:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334563968:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019021.329339:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334564096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019021.329352:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019021.329358:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019021/real 1734019021] req@ffff8800a78a3480 x1818247334564096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019021.329375:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334564096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019021.329382:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019021.329384:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019021.329387:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334564096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019021.329407:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334564224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019021.329420:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019021.329426:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019021/real 1734019021] req@ffff8800a78a0700 x1818247334564224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019021.329436:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334564224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019021.329444:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019021.329446:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019021.329449:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334564224:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019021.384744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019022.386779:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019023.388735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019024.390758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019025.392737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019026.336616:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019026.336625:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019026.336629:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019026.336639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3292 00000100:00080000:2.0:1734019026.336649:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019026.336656:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019026.336694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334564352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019026.336727:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019026.336731:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019026.336733:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019026.336737:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3292 00000100:00080000:2.0:1734019026.336740:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019026.336744:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019026.336754:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019026.336765:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334564480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019026.336774:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019026.336777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019026.336779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019026.336783:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3292 00000100:00080000:2.0:1734019026.336787:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019026.336791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019026.336795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019026.336804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334564608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019026.336813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019026.336816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019026.336817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019026.336820:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3292 00000100:00080000:2.0:1734019026.336823:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019026.336826:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734019026.336829:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334564352:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019026.336830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019026.336838:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334564736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019026.336847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019026.336850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019026.336851:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019026.336855:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3292 00000100:00080000:2.0:1734019026.336857:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019026.336860:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019026.336864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019026.336871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334564864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019026.336886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019026.336889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019026.336890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734019026.336893:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019026.336894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3292 00000100:00080000:2.0:1734019026.336896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019026.336900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019026.336904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019026.336911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334564992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019026.336916:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019026/real 1734019026] req@ffff8800a78a0700 x1818247334564352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019026.336919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019026.336922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019026.336923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019026.336927:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3292 00000100:00080000:2.0:1734019026.336929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019026.336932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734019026.336936:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334564352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019026.336936:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019026.336944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334565120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019026.336946:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019026.336950:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019026.336956:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334564352:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734019026.336963:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334564480:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734019026.336980:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019026.336987:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019026/real 1734019026] req@ffff8800a78a3480 x1818247334564480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019026.336998:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334564480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019026.337006:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019026.337009:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019026.337013:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334564480:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019026.337018:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334564608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019026.337031:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019026.337038:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019026/real 1734019026] req@ffff8800a78a1f80 x1818247334564608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019026.337048:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334564608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019026.337055:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019026.337057:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019026.337060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334564608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019026.337118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334564736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019026.337139:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019026.337147:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019026/real 1734019026] req@ffff8800a78a0a80 x1818247334564736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019026.337158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334564736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019026.337166:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019026.337168:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019026.337171:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334564736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019026.337176:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334564864:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734019026.337188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019026.337194:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019026/real 1734019026] req@ffff8800a78a2680 x1818247334564864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019026.337204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334564864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019026.337212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019026.337214:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019026.337217:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334564864:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019026.337222:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334564992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019026.337234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019026.337240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019026/real 1734019026] req@ffff8800a78a1880 x1818247334564992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019026.337256:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334564992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019026.337264:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019026.337266:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019026.337269:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334564992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019026.337274:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334565120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019026.337286:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019026.337292:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019026/real 1734019026] req@ffff8800a78a3b80 x1818247334565120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019026.337302:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334565120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019026.337309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019026.337311:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019026.337314:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334565120:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019026.394742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019027.396710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019028.398741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019029.400712:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019030.402760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019031.344600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019031.344610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019031.344615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019031.344625:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3297 00000100:00080000:2.0:1734019031.344634:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019031.344643:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019031.344679:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334565248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019031.344712:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019031.344716:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019031.344718:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019031.344722:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3297 00000100:00080000:2.0:1734019031.344725:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019031.344729:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019031.344739:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019031.344750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334565376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019031.344759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019031.344762:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019031.344764:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019031.344768:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3297 00000100:00080000:2.0:1734019031.344771:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019031.344774:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019031.344779:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019031.344788:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334565504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019031.344797:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019031.344800:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019031.344802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019031.344805:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3297 00000100:00080000:2.0:1734019031.344808:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019031.344811:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734019031.344812:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334565248:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019031.344815:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019031.344823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334565632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019031.344832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019031.344835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019031.344836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019031.344839:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3297 00000100:00080000:2.0:1734019031.344842:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019031.344845:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019031.344849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019031.344856:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334565760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019031.344864:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019031.344867:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019031.344868:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019031.344872:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3297 00000100:00080000:2.0:1734019031.344876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019031.344879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019031.344883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019031.344891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334565888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019031.344898:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019031.344900:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334565376:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019031.344901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019031.344903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019031.344906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3297 00000100:00080000:2.0:1734019031.344909:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019031.344911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019031.344916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019031.344920:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334565504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019031.344924:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334566016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019031.344943:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334565632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019031.344958:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334565760:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019031.344973:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334565888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019031.345014:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334566016:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019031.404742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019032.406732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019033.408732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019034.350748:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019034.350795:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019034.350814:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019034.350850:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019034.350857:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019034.350864:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019034.350871:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019034.350877:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019034.350883:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734019034.350980:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019031/real 1734019034] req@ffff8800a78a0a80 x1818247334565632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019034.351010:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334565632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019034.351023:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019034.351027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019034.351034:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334565632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019034.351046:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019031/real 1734019034] req@ffff8800a78a1f80 x1818247334565760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019034.351064:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334565760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019034.351076:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019034.351080:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019034.351085:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334565760:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019034.351095:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019031/real 1734019034] req@ffff8800a78a3480 x1818247334565888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019034.351113:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334565888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019034.351125:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019034.351130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019034.351135:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334565888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019034.351144:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019031/real 1734019034] req@ffff8800a78a0700 x1818247334566016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019034.351162:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334566016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019034.351173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019034.351177:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019034.351183:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334566016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019034.351193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019031/real 1734019034] req@ffff8800a78a3b80 x1818247334565248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019034.351212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334565248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019034.351225:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019034.351229:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019034.351234:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334565248:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019034.351244:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019031/real 1734019034] req@ffff8800a78a1880 x1818247334565376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019034.351263:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334565376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019034.351284:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019034.351289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019034.351294:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334565376:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019034.351303:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019031/real 1734019034] req@ffff8800a78a2680 x1818247334565504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019034.351322:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334565504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019034.351333:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019034.351337:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019034.351342:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334565504:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019034.410755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019035.412724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019036.352570:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019036.352579:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019036.352583:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019036.352594:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3302 00000100:00080000:2.0:1734019036.352603:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019036.352611:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019036.352647:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334566144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019036.352680:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019036.352684:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019036.352686:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019036.352690:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3302 00000100:00080000:2.0:1734019036.352693:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019036.352697:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019036.352706:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019036.352717:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334566272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019036.352727:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019036.352730:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019036.352732:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019036.352735:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3302 00000100:00080000:2.0:1734019036.352738:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019036.352741:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019036.352746:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019036.352754:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334566400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019036.352763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019036.352766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019036.352768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019036.352772:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3302 00000100:00080000:2.0:1734019036.352774:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019036.352777:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019036.352781:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019036.352790:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334566144:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019036.352790:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334566528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019036.352798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019036.352801:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019036.352803:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019036.352806:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3302 00000100:00080000:2.0:1734019036.352809:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019036.352812:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019036.352816:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019036.352823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334566656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019036.352831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019036.352834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019036.352836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019036.352839:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3302 00000100:00080000:2.0:1734019036.352851:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019036.352855:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019036.352856:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019036.352859:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019036.352867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334566784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019036.352876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019036.352877:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019036/real 1734019036] req@ffff8800a78a2680 x1818247334566144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019036.352880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019036.352881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019036.352885:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3302 00000100:00080000:2.0:1734019036.352887:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019036.352890:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019036.352895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019036.352896:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334566144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019036.352925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334566912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019036.352933:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019036.352937:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019036.352942:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334566144:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019036.352950:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334566272:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019036.352969:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019036.352978:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019036/real 1734019036] req@ffff8800a78a1880 x1818247334566272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019036.352989:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334566272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019036.352997:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019036.353000:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019036.353004:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334566272:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019036.353009:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334566400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019036.353023:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019036.353029:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019036/real 1734019036] req@ffff8800a78a3b80 x1818247334566400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019036.353039:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334566400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019036.353046:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019036.353049:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019036.353052:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334566400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019036.353075:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334566528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019036.353088:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019036.353094:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019036/real 1734019036] req@ffff8800a78a0700 x1818247334566528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019036.353104:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334566528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019036.353119:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019036.353121:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019036.353124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334566528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019036.353130:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334566656:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019036.353142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019036.353148:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019036/real 1734019036] req@ffff8800a78a3480 x1818247334566656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019036.353158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334566656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019036.353165:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019036.353167:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019036.353170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334566656:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019036.353176:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334566784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019036.353188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019036.353193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019036/real 1734019036] req@ffff8800a78a1f80 x1818247334566784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019036.353203:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334566784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019036.353210:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019036.353212:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019036.353215:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334566784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019036.353221:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334566912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019036.353234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019036.353240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019036/real 1734019036] req@ffff8800a78a0a80 x1818247334566912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019036.353250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334566912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019036.353257:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019036.353259:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019036.353261:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334566912:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019036.414716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019037.416671:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019038.418760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019039.420807:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019040.422710:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019041.360605:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019041.360614:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019041.360618:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019041.360629:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3307 00000100:00080000:2.0:1734019041.360638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019041.360646:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019041.360682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334567040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019041.360715:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019041.360719:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019041.360722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019041.360726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3307 00000100:00080000:2.0:1734019041.360729:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019041.360733:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019041.360742:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019041.360754:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334567168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019041.360763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019041.360767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019041.360768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019041.360772:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3307 00000100:00080000:2.0:1734019041.360775:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019041.360779:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019041.360783:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019041.360792:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334567296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019041.360800:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019041.360803:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019041.360805:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019041.360808:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3307 00000100:00080000:2.0:1734019041.360810:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019041.360844:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019041.360849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019041.360858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334567424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019041.360860:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334567040:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019041.360869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019041.360872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019041.360874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019041.360877:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3307 00000100:00080000:2.0:1734019041.360880:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019041.360883:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019041.360887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019041.360894:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334567552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019041.360903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019041.360906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019041.360907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019041.360910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3307 00000100:00080000:2.0:1734019041.360915:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019041.360918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019041.360922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019041.360925:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019041.360929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334567680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019041.360937:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019041.360940:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019041.360942:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019041.360945:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3307 00000100:00100000:1.0:1734019041.360946:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019041/real 1734019041] req@ffff8800a78a0a80 x1818247334567040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019041.360949:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019041.360952:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019041.360956:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019041.360963:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334567808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019041.360965:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334567040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019041.360975:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019041.360979:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019041.360983:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334567040:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019041.360990:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334567168:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019041.361007:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019041.361015:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019041/real 1734019041] req@ffff8800a78a1f80 x1818247334567168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019041.361034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334567168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019041.361042:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019041.361044:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019041.361048:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334567168:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019041.361054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334567296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019041.361067:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019041.361073:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019041/real 1734019041] req@ffff8800a78a3480 x1818247334567296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019041.361083:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334567296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019041.361090:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019041.361092:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019041.361095:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334567296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019041.361120:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334567424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019041.361134:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019041.361140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019041/real 1734019041] req@ffff8800a78a0700 x1818247334567424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019041.361150:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334567424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019041.361158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019041.361160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019041.361164:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334567424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019041.361169:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334567552:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019041.361182:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019041.361188:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019041/real 1734019041] req@ffff8800a78a3b80 x1818247334567552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019041.361198:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334567552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019041.361205:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019041.361207:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019041.361210:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334567552:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019041.361215:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334567680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019041.361227:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019041.361232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019041/real 1734019041] req@ffff8800a78a1880 x1818247334567680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019041.361242:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334567680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019041.361249:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019041.361251:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019041.361254:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334567680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019041.361259:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334567808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019041.361274:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019041.361280:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019041/real 1734019041] req@ffff8800a78a2680 x1818247334567808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019041.361290:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334567808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019041.361297:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019041.361299:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019041.361302:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334567808:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019041.424726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019042.426719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019043.428757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019044.430722:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019045.432790:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019046.368611:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019046.368620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019046.368624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019046.368635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3313 00000100:00080000:2.0:1734019046.368643:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019046.368651:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019046.368688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334567936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019046.368723:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019046.368749:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019046.368752:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019046.368756:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3313 00000100:00080000:2.0:1734019046.368760:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019046.368764:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019046.368774:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019046.368786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334568064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019046.368796:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019046.368800:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019046.368801:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019046.368805:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3313 00000100:00080000:2.0:1734019046.368807:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019046.368810:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019046.368815:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019046.368823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334568192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019046.368833:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334567936:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019046.368833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019046.368836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019046.368837:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019046.368841:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3313 00000100:00080000:2.0:1734019046.368843:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019046.368846:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019046.368850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019046.368858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334568320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019046.368866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019046.368869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019046.368871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019046.368875:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3313 00000100:00080000:2.0:1734019046.368878:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019046.368881:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019046.368885:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019046.368892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334568448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019046.368899:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019046.368900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019046.368903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019046.368904:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019046.368908:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3313 00000100:00080000:2.0:1734019046.368910:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019046.368914:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019046.368917:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019046.368920:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019046/real 1734019046] req@ffff8800a78a2680 x1818247334567936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019046.368925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334568576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019046.368932:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019046.368935:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019046.368937:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019046.368939:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334567936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019046.368940:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3313 00000100:00080000:2.0:1734019046.368943:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019046.368946:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734019046.368950:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1734019046.368950:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734019046.368954:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019046.368957:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334568704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019046.368959:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334567936:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019046.368966:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334568064:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019046.368983:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019046.368989:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019046/real 1734019046] req@ffff8800a78a1880 x1818247334568064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019046.369001:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334568064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019046.369009:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019046.369012:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019046.369016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334568064:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019046.369037:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334568192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019046.369051:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019046.369057:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019046/real 1734019046] req@ffff8800a78a3b80 x1818247334568192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019046.369068:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334568192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019046.369075:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019046.369078:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019046.369081:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334568192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019046.369086:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334568320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019046.369098:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019046.369104:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019046/real 1734019046] req@ffff8800a78a0700 x1818247334568320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019046.369114:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334568320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019046.369121:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019046.369123:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019046.369126:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334568320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019046.369132:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334568448:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019046.369144:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019046.369150:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019046/real 1734019046] req@ffff8800a78a3480 x1818247334568448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019046.369161:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334568448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019046.369168:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019046.369170:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019046.369173:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334568448:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019046.369178:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334568576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019046.369190:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019046.369196:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019046/real 1734019046] req@ffff8800a78a1f80 x1818247334568576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019046.369206:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334568576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019046.369213:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019046.369215:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019046.369218:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334568576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019046.369223:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334568704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019046.369235:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019046.369240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019046/real 1734019046] req@ffff8800a78a0a80 x1818247334568704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019046.369251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334568704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019046.369259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019046.369261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019046.369264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334568704:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019046.434700:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019047.436779:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019048.438729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019049.440750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019050.442737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019051.376584:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019051.376598:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019051.376604:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019051.376617:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3318 00000100:00080000:2.0:1734019051.376629:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019051.376639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019051.376730:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334568832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019051.376755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019051.376757:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019051.376758:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019051.376761:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3318 00000100:00080000:2.0:1734019051.376764:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019051.376766:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019051.376787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019051.376795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334568960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019051.376802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019051.376804:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019051.376806:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019051.376809:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3318 00000100:00080000:2.0:1734019051.376811:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019051.376814:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019051.376817:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019051.376823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334569088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019051.376829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019051.376831:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019051.376832:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019051.376834:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3318 00000100:00080000:2.0:1734019051.376836:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019051.376838:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019051.376841:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334568832:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019051.376841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019051.376847:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334569216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019051.376852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019051.376854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019051.376854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019051.376856:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3318 00000100:00080000:2.0:1734019051.376858:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019051.376860:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019051.376863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019051.376867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334569344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019051.376873:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019051.376876:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019051.376877:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019051.376879:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3318 00000100:00080000:2.0:1734019051.376880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019051.376883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019051.376885:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019051.376890:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334569472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019051.376894:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019051.376895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019051.376897:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019051.376898:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019051.376900:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3318 00000100:00080000:2.0:1734019051.376902:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019051.376904:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019051.376907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019051.376910:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019051/real 1734019051] req@ffff8800a78a0a80 x1818247334568832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019051.376912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334569600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019051.376921:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334568832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019051.376928:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019051.376930:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019051.376932:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334568832:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019051.376937:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334568960:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019051.376947:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019051.376951:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019051/real 1734019051] req@ffff8800a78a1f80 x1818247334568960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019051.376956:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334568960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019051.376961:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019051.376963:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019051.376965:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334568960:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019051.376980:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334569088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019051.376988:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019051.376991:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019051/real 1734019051] req@ffff8800a78a3480 x1818247334569088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019051.376997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334569088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019051.377001:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019051.377002:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019051.377003:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334569088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019051.377007:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334569216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019051.377013:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019051.377016:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019051/real 1734019051] req@ffff8800a78a0700 x1818247334569216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019051.377021:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334569216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019051.377025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019051.377026:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019051.377028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334569216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019051.377031:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334569344:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019051.377040:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019051.377050:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019051/real 1734019051] req@ffff8800a78a3b80 x1818247334569344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019051.377056:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334569344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019051.377060:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019051.377061:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019051.377062:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334569344:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019051.377065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334569472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019051.377072:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019051.377075:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019051/real 1734019051] req@ffff8800a78a1880 x1818247334569472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019051.377080:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334569472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019051.377084:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019051.377085:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019051.377087:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334569472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019051.377090:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334569600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019051.377097:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019051.377100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019051/real 1734019051] req@ffff8800a78a2680 x1818247334569600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019051.377109:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334569600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019051.377113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019051.377114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019051.377115:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334569600:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019051.444639:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019052.446612:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019053.448710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019054.450792:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019055.452827:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019056.384530:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019056.384535:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019056.384538:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019056.384544:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3323 00000100:00080000:2.0:1734019056.384549:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019056.384554:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019056.384591:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334569728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019056.384608:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019056.384610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019056.384611:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019056.384614:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3323 00000100:00080000:2.0:1734019056.384616:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019056.384618:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019056.384623:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019056.384629:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334569856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019056.384635:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019056.384636:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019056.384637:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019056.384639:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3323 00000100:00080000:2.0:1734019056.384640:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019056.384642:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019056.384645:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334569728:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019056.384645:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019056.384650:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334569984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019056.384655:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019056.384657:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019056.384658:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019056.384660:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3323 00000100:00080000:2.0:1734019056.384661:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019056.384663:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019056.384665:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019056.384669:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334570112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019056.384673:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019056.384675:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019056.384675:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019056.384677:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3323 00000100:00080000:2.0:1734019056.384679:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019056.384680:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019056.384682:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019056.384687:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334570240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019056.384689:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019056.384691:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019056.384692:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019056.384693:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019056.384695:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3323 00000100:00080000:2.0:1734019056.384697:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019056.384698:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019056.384701:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019056.384702:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019056/real 1734019056] req@ffff8800a78a2680 x1818247334569728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019056.384705:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334570368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019056.384709:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019056.384711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019056.384711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019056.384713:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334569728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019056.384713:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3323 00000100:00080000:2.0:1734019056.384715:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019056.384716:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734019056.384719:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1734019056.384719:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734019056.384721:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019056.384722:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334570496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019056.384724:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334569728:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019056.384728:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334569856:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019056.384738:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019056.384742:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019056/real 1734019056] req@ffff8800a78a1880 x1818247334569856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019056.384748:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334569856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019056.384752:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019056.384754:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019056.384756:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334569856:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019056.384770:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334569984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019056.384778:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019056.384781:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019056/real 1734019056] req@ffff8800a78a3b80 x1818247334569984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019056.384788:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334569984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019056.384791:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019056.384793:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019056.384794:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334569984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019056.384797:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334570112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019056.384803:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019056.384806:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019056/real 1734019056] req@ffff8800a78a0700 x1818247334570112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019056.384811:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334570112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019056.384815:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019056.384816:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019056.384818:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334570112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019056.384821:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334570240:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019056.384829:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019056.384833:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019056/real 1734019056] req@ffff8800a78a3480 x1818247334570240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019056.384838:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334570240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019056.384842:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019056.384843:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019056.384844:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334570240:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019056.384847:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334570368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019056.384854:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019056.384857:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019056/real 1734019056] req@ffff8800a78a1f80 x1818247334570368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019056.384862:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334570368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019056.384865:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019056.384866:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019056.384868:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334570368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019056.384871:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334570496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019056.384878:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019056.384881:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019056/real 1734019056] req@ffff8800a78a0a80 x1818247334570496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019056.384890:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334570496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019056.384894:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019056.384895:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019056.384896:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334570496:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019056.454755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019057.456808:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019058.458802:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019059.460793:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019060.462727:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019061.392595:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019061.392605:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019061.392609:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019061.392619:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3328 00000100:00080000:2.0:1734019061.392628:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019061.392635:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019061.392674:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334570624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019061.392708:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019061.392712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019061.392714:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019061.392718:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3328 00000100:00080000:2.0:1734019061.392721:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019061.392725:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019061.392734:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019061.392746:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334570752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019061.392755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019061.392758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019061.392760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019061.392764:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3328 00000100:00080000:2.0:1734019061.392766:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019061.392770:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019061.392774:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019061.392782:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334570880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019061.392791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019061.392794:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019061.392796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019061.392800:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3328 00000100:00080000:2.0:1734019061.392802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019061.392805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019061.392809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019061.392817:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334570624:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019061.392817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334571008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019061.392826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019061.392829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019061.392830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019061.392833:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3328 00000100:00080000:2.0:1734019061.392836:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019061.392839:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019061.392843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019061.392850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334571136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019061.392858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019061.392860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019061.392862:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019061.392865:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3328 00000100:00080000:2.0:1734019061.392877:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019061.392880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019061.392881:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019061.392884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019061.392892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334571264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019061.392902:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019061/real 1734019061] req@ffff8800a78a0a80 x1818247334570624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019061.392902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019061.392905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019061.392906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019061.392910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3328 00000100:00080000:2.0:1734019061.392912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019061.392915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019061.392919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019061.392921:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334570624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019061.392927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334571392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019061.392932:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019061.392935:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019061.392940:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334570624:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019061.392947:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334570752:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019061.392964:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019061.392970:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019061/real 1734019061] req@ffff8800a78a1f80 x1818247334570752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019061.392981:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334570752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019061.392989:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019061.392992:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019061.392996:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334570752:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019061.393002:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334570880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019061.393016:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019061.393022:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019061/real 1734019061] req@ffff8800a78a3480 x1818247334570880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019061.393032:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334570880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019061.393040:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019061.393042:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019061.393046:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334570880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019061.393070:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334571008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019061.393083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019061.393089:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019061/real 1734019061] req@ffff8800a78a0700 x1818247334571008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019061.393099:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334571008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019061.393107:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019061.393109:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019061.393112:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334571008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019061.393118:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334571136:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019061.393133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019061.393139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019061/real 1734019061] req@ffff8800a78a3b80 x1818247334571136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019061.393149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334571136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019061.393156:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019061.393158:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019061.393161:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334571136:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019061.393167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334571264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019061.393178:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019061.393184:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019061/real 1734019061] req@ffff8800a78a1880 x1818247334571264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019061.393194:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334571264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019061.393201:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019061.393203:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019061.393206:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334571264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019061.393211:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334571392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019061.393223:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019061.393229:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019061/real 1734019061] req@ffff8800a78a2680 x1818247334571392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019061.393246:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334571392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019061.393253:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019061.393255:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019061.393258:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334571392:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019061.464751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019062.466727:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019063.468738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019064.470745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019065.472734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019066.400578:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019066.400588:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019066.400592:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019066.400602:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3333 00000100:00080000:2.0:1734019066.400612:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019066.400619:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019066.400658:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334571520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019066.400692:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019066.400696:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019066.400699:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019066.400703:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3333 00000100:00080000:2.0:1734019066.400706:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019066.400710:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019066.400719:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019066.400730:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334571648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019066.400740:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019066.400743:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019066.400745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019066.400748:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3333 00000100:00080000:2.0:1734019066.400751:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019066.400754:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019066.400759:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019066.400767:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334571776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019066.400776:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019066.400779:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019066.400780:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019066.400783:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3333 00000100:00080000:2.0:1734019066.400786:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019066.400789:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019066.400793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019066.400801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334571904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019066.400811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019066.400814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019066.400816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019066.400819:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3333 00000100:00100000:1.0:1734019066.400821:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334571520:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019066.400822:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019066.400825:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019066.400829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019066.400836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334572032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019066.400844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019066.400847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019066.400848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019066.400852:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3333 00000100:00080000:2.0:1734019066.400854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019066.400857:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019066.400861:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019066.400869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334572160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019066.400876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019066.400879:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019066.400881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019066.400884:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3333 00000800:00000100:1.0:1734019066.400887:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019066.400887:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019066.400891:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019066.400895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019066.400902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334572288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019066.400908:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019066/real 1734019066] req@ffff8800a78a2680 x1818247334571520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019066.400927:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334571520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019066.400938:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019066.400942:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019066.400947:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334571520:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019066.400954:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334571648:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019066.400970:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019066.400977:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019066/real 1734019066] req@ffff8800a78a1880 x1818247334571648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019066.400988:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334571648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019066.400996:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019066.401000:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019066.401003:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334571648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019066.401009:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334571776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019066.401023:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019066.401029:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019066/real 1734019066] req@ffff8800a78a3b80 x1818247334571776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019066.401039:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334571776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019066.401047:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019066.401049:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019066.401053:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334571776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019066.401076:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334571904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019066.401089:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019066.401095:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019066/real 1734019066] req@ffff8800a78a0700 x1818247334571904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019066.401105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334571904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019066.401113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019066.401115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019066.401118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334571904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019066.401124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334572032:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019066.401139:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019066.401146:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019066/real 1734019066] req@ffff8800a78a3480 x1818247334572032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019066.401156:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334572032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019066.401164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019066.401166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019066.401169:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334572032:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019066.401174:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334572160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019066.401186:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019066.401192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019066/real 1734019066] req@ffff8800a78a1f80 x1818247334572160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019066.401202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334572160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019066.401209:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019066.401211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019066.401214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334572160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019066.401219:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334572288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019066.401231:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019066.401237:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019066/real 1734019066] req@ffff8800a78a0a80 x1818247334572288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019066.401253:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334572288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019066.401260:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019066.401262:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019066.401265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334572288:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019066.474723:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019067.476750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019068.478721:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019069.480739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019070.482745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019071.408586:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019071.408599:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019071.408605:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019071.408619:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3338 00000100:00080000:2.0:1734019071.408630:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019071.408640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019071.408690:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334572416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019071.408731:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019071.408735:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019071.408737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019071.408742:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3338 00000100:00080000:2.0:1734019071.408746:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019071.408749:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019071.408761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019071.408774:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334572544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019071.408785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019071.408788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019071.408790:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019071.408794:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3338 00000100:00080000:2.0:1734019071.408796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019071.408800:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019071.408805:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019071.408815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334572672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019071.408824:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019071.408828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019071.408829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019071.408833:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3338 00000100:00080000:2.0:1734019071.408835:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019071.408839:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019071.408843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019071.408852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334572800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019071.408863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019071.408865:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334572416:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019071.408866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019071.408867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019071.408873:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3338 00000100:00080000:2.0:1734019071.408875:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019071.408879:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019071.408883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019071.408892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334572928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019071.408900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019071.408903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019071.408905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019071.408908:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3338 00000100:00080000:2.0:1734019071.408911:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019071.408914:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019071.408918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019071.408927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334573056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019071.408934:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734019071.408936:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019071.408938:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019071.408939:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019071.408943:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3338 00000100:00080000:2.0:1734019071.408946:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019071.408950:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019071.408954:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019071.408958:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019071/real 1734019071] req@ffff8800a78a0a80 x1818247334572416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019071.408962:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334573184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019071.408977:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334572416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019071.408989:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019071.408993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019071.408998:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334572416:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019071.409006:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334572544:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019071.409025:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019071.409032:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019071/real 1734019071] req@ffff8800a78a1f80 x1818247334572544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019071.409043:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334572544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019071.409051:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019071.409055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019071.409058:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334572544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019071.409065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334572672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019071.409079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019071.409085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019071/real 1734019071] req@ffff8800a78a3480 x1818247334572672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019071.409096:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334572672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019071.409104:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019071.409106:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019071.409110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334572672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019071.409137:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334572800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019071.409151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019071.409157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019071/real 1734019071] req@ffff8800a78a0700 x1818247334572800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019071.409168:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334572800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019071.409176:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019071.409178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019071.409181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334572800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019071.409187:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334572928:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019071.409204:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019071.409210:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019071/real 1734019071] req@ffff8800a78a3b80 x1818247334572928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019071.409220:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334572928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019071.409228:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019071.409230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019071.409233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334572928:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019071.409239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334573056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019071.409251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019071.409257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019071/real 1734019071] req@ffff8800a78a1880 x1818247334573056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019071.409267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334573056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019071.409274:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019071.409276:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019071.409279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334573056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019071.409285:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334573184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019071.409297:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019071.409346:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019071/real 1734019071] req@ffff8800a78a2680 x1818247334573184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019071.409365:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334573184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019071.409373:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019071.409375:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019071.409378:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334573184:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019071.484747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019072.486692:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019073.488762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019074.490760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019075.492758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019076.416601:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019076.416610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019076.416615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019076.416625:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3343 00000100:00080000:2.0:1734019076.416633:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019076.416641:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019076.416678:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334573312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019076.416712:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019076.416716:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019076.416718:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019076.416722:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3343 00000100:00080000:2.0:1734019076.416726:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019076.416729:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019076.416738:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019076.416750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334573440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019076.416760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019076.416763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019076.416766:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019076.416770:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3343 00000100:00080000:2.0:1734019076.416772:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019076.416776:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019076.416780:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019076.416789:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334573568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019076.416798:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019076.416801:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019076.416803:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019076.416806:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3343 00000100:00080000:2.0:1734019076.416808:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019076.416811:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019076.416815:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019076.416824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334573696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019076.416825:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334573312:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019076.416833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019076.416836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019076.416838:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019076.416841:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3343 00000100:00080000:2.0:1734019076.416844:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019076.416847:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019076.416850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019076.416858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334573824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019076.416867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019076.416870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019076.416871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019076.416874:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3343 00000100:00080000:2.0:1734019076.416886:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019076.416890:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019076.416890:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019076.416894:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019076.416902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334573952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019076.416910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019076.416911:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019076/real 1734019076] req@ffff8800a78a2680 x1818247334573312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019076.416913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019076.416915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019076.416918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3343 00000100:00080000:2.0:1734019076.416921:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019076.416924:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019076.416928:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019076.416931:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334573312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019076.416935:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334574080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019076.416941:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019076.416944:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019076.416949:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334573312:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019076.416957:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334573440:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019076.416973:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019076.416980:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019076/real 1734019076] req@ffff8800a78a1880 x1818247334573440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019076.416991:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334573440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019076.416999:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019076.417002:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019076.417006:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334573440:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019076.417012:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334573568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019076.417026:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019076.417032:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019076/real 1734019076] req@ffff8800a78a3b80 x1818247334573568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019076.417042:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334573568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019076.417050:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019076.417052:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019076.417056:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334573568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019076.417079:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334573696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019076.417092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019076.417098:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019076/real 1734019076] req@ffff8800a78a0700 x1818247334573696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019076.417108:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334573696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019076.417117:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019076.417119:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019076.417122:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334573696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019076.417127:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334573824:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019076.417142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019076.417149:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019076/real 1734019076] req@ffff8800a78a3480 x1818247334573824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019076.417159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334573824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019076.417167:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019076.417169:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019076.417171:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334573824:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019076.417177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334573952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019076.417188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019076.417194:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019076/real 1734019076] req@ffff8800a78a1f80 x1818247334573952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019076.417204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334573952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019076.417211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019076.417213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019076.417216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334573952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019076.417268:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334574080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019076.417285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019076.417292:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019076/real 1734019076] req@ffff8800a78a0a80 x1818247334574080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019076.417309:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334574080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019076.417317:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019076.417319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019076.417322:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334574080:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019076.494728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019077.496730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019078.498738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019079.500750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019080.502717:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019081.424606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019081.424616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019081.424621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019081.424632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3348 00000100:00080000:2.0:1734019081.424641:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019081.424648:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019081.424685:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334574208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019081.424719:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019081.424724:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019081.424725:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019081.424730:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3348 00000100:00080000:2.0:1734019081.424733:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019081.424737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019081.424746:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019081.424758:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334574336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019081.424767:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019081.424771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019081.424773:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019081.424777:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3348 00000100:00080000:2.0:1734019081.424779:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019081.424783:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019081.424787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019081.424795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334574464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019081.424805:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019081.424808:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019081.424810:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019081.424813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3348 00000100:00080000:2.0:1734019081.424816:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019081.424819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019081.424823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019081.424831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334574592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019081.424832:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334574208:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019081.424840:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019081.424843:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019081.424844:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019081.424847:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3348 00000100:00080000:2.0:1734019081.424850:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019081.424853:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019081.424857:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019081.424864:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334574720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019081.424872:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019081.424875:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019081.424876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019081.424879:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3348 00000100:00080000:2.0:1734019081.424883:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019081.424886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019081.424890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019081.424898:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019081.424898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334574848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019081.424906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019081.424909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019081.424910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019081.424913:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3348 00000100:00080000:2.0:1734019081.424916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019081.424919:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019081/real 1734019081] req@ffff8800a78a0a80 x1818247334574208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019081.424919:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019081.424923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019081.424931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334574976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019081.424938:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334574208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019081.424948:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019081.424952:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019081.424957:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334574208:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019081.424964:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334574336:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019081.424981:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019081.424987:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019081/real 1734019081] req@ffff8800a78a1f80 x1818247334574336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019081.424998:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334574336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019081.425007:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019081.425010:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019081.425013:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334574336:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019081.425019:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334574464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019081.425033:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019081.425039:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019081/real 1734019081] req@ffff8800a78a3480 x1818247334574464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019081.425049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334574464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019081.425057:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019081.425059:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019081.425063:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334574464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019081.425087:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334574592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019081.425100:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019081.425106:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019081/real 1734019081] req@ffff8800a78a0700 x1818247334574592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019081.425117:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334574592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019081.425124:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019081.425127:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019081.425129:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334574592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019081.425189:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334574720:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019081.425211:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019081.425219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019081/real 1734019081] req@ffff8800a78a3b80 x1818247334574720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019081.425230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334574720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019081.425238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019081.425241:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019081.425244:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334574720:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019081.425249:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334574848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019081.425261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019081.425268:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019081/real 1734019081] req@ffff8800a78a1880 x1818247334574848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019081.425278:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334574848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019081.425285:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019081.425287:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019081.425290:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334574848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019081.425296:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334574976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019081.425308:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019081.425314:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019081/real 1734019081] req@ffff8800a78a2680 x1818247334574976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019081.425330:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334574976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019081.425338:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019081.425340:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019081.425343:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334574976:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019081.504736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019082.506729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019083.508741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019084.510752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019085.512741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019086.432599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019086.432608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019086.432612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019086.432622:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3353 00000100:00080000:2.0:1734019086.432631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019086.432639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019086.432677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334575104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019086.432712:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019086.432716:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019086.432718:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019086.432722:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3353 00000100:00080000:2.0:1734019086.432725:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019086.432729:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019086.432738:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019086.432750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334575232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019086.432760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019086.432763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019086.432765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019086.432769:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3353 00000100:00080000:2.0:1734019086.432771:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019086.432775:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019086.432780:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019086.432788:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334575360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019086.432797:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019086.432800:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019086.432802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019086.432806:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3353 00000100:00080000:2.0:1734019086.432808:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019086.432811:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019086.432815:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019086.432822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334575104:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019086.432824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334575488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019086.432833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019086.432836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019086.432837:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019086.432840:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3353 00000100:00080000:2.0:1734019086.432843:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019086.432846:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019086.432850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019086.432857:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334575616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019086.432865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019086.432868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019086.432869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019086.432873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3353 00000100:00080000:2.0:1734019086.432877:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019086.432880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019086.432884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019086.432887:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019086.432892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334575744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019086.432901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019086.432904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019086.432905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019086.432909:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019086/real 1734019086] req@ffff8800a78a2680 x1818247334575104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019086.432909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3353 00000100:00080000:2.0:1734019086.432912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019086.432915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019086.432919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019086.432926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334575872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019086.432929:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334575104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019086.432940:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019086.432943:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019086.432947:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334575104:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019086.432955:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334575232:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019086.432972:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019086.432979:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019086/real 1734019086] req@ffff8800a78a1880 x1818247334575232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019086.432990:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334575232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019086.432998:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019086.433001:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019086.433005:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334575232:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019086.433011:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334575360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019086.433024:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019086.433030:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019086/real 1734019086] req@ffff8800a78a3b80 x1818247334575360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019086.433041:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334575360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019086.433091:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019086.433094:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019086.433098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334575360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019086.433123:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334575488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019086.433141:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019086.433148:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019086/real 1734019086] req@ffff8800a78a0700 x1818247334575488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019086.433159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334575488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019086.433167:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019086.433169:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019086.433172:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334575488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019086.433177:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334575616:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019086.433193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019086.433199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019086/real 1734019086] req@ffff8800a78a3480 x1818247334575616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019086.433210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334575616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019086.433217:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019086.433220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019086.433223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334575616:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019086.433228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334575744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019086.433239:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019086.433245:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019086/real 1734019086] req@ffff8800a78a1f80 x1818247334575744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019086.433255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334575744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019086.433262:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019086.433264:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019086.433267:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334575744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019086.433272:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334575872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019086.433285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019086.433291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019086/real 1734019086] req@ffff8800a78a0a80 x1818247334575872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019086.433307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334575872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019086.433314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019086.433316:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019086.433319:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334575872:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019086.514714:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019087.516764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019088.518715:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019089.520781:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019090.522761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019091.440611:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019091.440620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019091.440624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019091.440635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3358 00000100:00080000:2.0:1734019091.440644:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019091.440651:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019091.440688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334576000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019091.440722:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019091.440726:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019091.440729:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019091.440733:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3358 00000100:00080000:2.0:1734019091.440736:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019091.440740:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019091.440749:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019091.440760:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334576128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019091.440770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019091.440773:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019091.440775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019091.440779:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3358 00000100:00080000:2.0:1734019091.440781:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019091.440784:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019091.440789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019091.440797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334576256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019091.440806:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019091.440809:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019091.440811:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019091.440814:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3358 00000100:00080000:2.0:1734019091.440816:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019091.440820:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019091.440823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019091.440832:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334576384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019091.440834:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334576000:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019091.440842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019091.440845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019091.440846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019091.440850:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3358 00000100:00080000:2.0:1734019091.440852:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019091.440855:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019091.440859:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019091.440866:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334576512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019091.440874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019091.440877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019091.440878:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019091.440882:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3358 00000100:00080000:2.0:1734019091.440894:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019091.440898:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019091.440900:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019091.440902:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019091.440910:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334576640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019091.440918:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019091.440921:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019091/real 1734019091] req@ffff8800a78a0a80 x1818247334576000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019091.440921:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019091.440922:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019091.440926:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3358 00000100:00080000:2.0:1734019091.440929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019091.440932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019091.440936:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019091.440940:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334576000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019091.440944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334576768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019091.440951:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019091.440954:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019091.440959:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334576000:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019091.441002:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334576128:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019091.441024:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019091.441032:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019091/real 1734019091] req@ffff8800a78a1f80 x1818247334576128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019091.441044:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334576128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019091.441052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019091.441055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019091.441059:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334576128:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019091.441065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334576256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019091.441079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019091.441085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019091/real 1734019091] req@ffff8800a78a3480 x1818247334576256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019091.441096:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334576256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019091.441103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019091.441106:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019091.441109:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334576256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019091.441138:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334576384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019091.441151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019091.441158:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019091/real 1734019091] req@ffff8800a78a0700 x1818247334576384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019091.441168:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334576384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019091.441176:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019091.441178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019091.441181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334576384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019091.441187:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334576512:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019091.441202:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019091.441209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019091/real 1734019091] req@ffff8800a78a3b80 x1818247334576512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019091.441219:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334576512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019091.441226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019091.441228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019091.441231:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334576512:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019091.441237:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334576640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019091.441248:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019091.441255:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019091/real 1734019091] req@ffff8800a78a1880 x1818247334576640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019091.441264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334576640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019091.441271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019091.441273:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019091.441277:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334576640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019091.441282:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334576768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019091.441295:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019091.441301:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019091/real 1734019091] req@ffff8800a78a2680 x1818247334576768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019091.441317:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334576768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019091.441324:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019091.441326:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019091.441329:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334576768:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019091.524712:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019092.526746:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019093.528761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019094.530737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019095.532738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019096.448596:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019096.448605:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019096.448609:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019096.448619:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3363 00000100:00080000:2.0:1734019096.448629:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019096.448637:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019096.448672:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334576896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019096.448705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019096.448709:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019096.448710:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019096.448715:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3363 00000100:00080000:2.0:1734019096.448718:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019096.448722:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019096.448731:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019096.448743:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334577024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019096.448753:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019096.448756:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019096.448758:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019096.448762:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3363 00000100:00080000:2.0:1734019096.448764:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019096.448768:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019096.448772:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019096.448781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334577152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019096.448790:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019096.448793:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019096.448794:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019096.448797:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3363 00000100:00080000:2.0:1734019096.448800:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019096.448803:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019096.448807:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019096.448811:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334576896:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019096.448815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334577280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019096.448825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019096.448828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019096.448829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019096.448834:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3363 00000100:00080000:2.0:1734019096.448837:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019096.448840:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019096.448844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019096.448852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334577408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019096.448859:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019096.448863:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019096.448864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019096.448868:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3363 00000100:00080000:2.0:1734019096.448870:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019096.448873:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019096.448911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019096.448922:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334577536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019096.448924:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334577024:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019096.448930:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019096.448933:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019096.448935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019096.448938:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3363 00000100:00080000:2.0:1734019096.448941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019096.448945:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019096.448947:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334577152:192.168.201.116@tcp:8:kworker.0 00000008:00100000:2.0:1734019096.448949:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019096.448957:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334577664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019096.448970:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334577280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019096.448985:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334577408:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019096.448999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334577536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019096.449013:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334577664:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019096.534718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019097.536731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019098.538729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019099.454767:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019099.454861:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019099.454884:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019099.454936:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019099.454948:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019099.454959:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019099.454970:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019099.454981:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019099.454991:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734019099.455063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019096/real 1734019099] req@ffff8800a78a0700 x1818247334577280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019099.455092:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334577280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019099.455104:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019099.455108:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019099.455114:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334577280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019099.455122:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019096/real 1734019099] req@ffff8800a78a3480 x1818247334577408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019099.455135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334577408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019099.455142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019099.455145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019099.455149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334577408:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019099.455157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019096/real 1734019099] req@ffff8800a78a1f80 x1818247334577536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019099.455175:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334577536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019099.455186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019099.455189:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019099.455194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334577536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019099.455204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019096/real 1734019099] req@ffff8800a78a0a80 x1818247334577664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019099.455219:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334577664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019099.455231:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019099.455235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019099.455239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334577664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019099.455253:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019096/real 1734019099] req@ffff8800a78a2680 x1818247334576896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019099.455272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334576896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019099.455283:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019099.455287:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019099.455291:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334576896:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019099.455301:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019096/real 1734019099] req@ffff8800a78a1880 x1818247334577024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019099.455319:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334577024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019099.455331:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019099.455335:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019099.455340:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334577024:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019099.455350:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019096/real 1734019099] req@ffff8800a78a3b80 x1818247334577152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019099.455367:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334577152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019099.455380:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019099.455384:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019099.455389:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334577152:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019099.540736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019100.542734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019101.456592:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019101.456602:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019101.456607:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019101.456617:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3368 00000100:00080000:2.0:1734019101.456626:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019101.456634:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019101.456671:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334577792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019101.456705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019101.456709:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019101.456711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019101.456715:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3368 00000100:00080000:2.0:1734019101.456718:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019101.456722:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019101.456731:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019101.456743:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334577920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019101.456752:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019101.456755:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019101.456758:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019101.456762:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3368 00000100:00080000:2.0:1734019101.456765:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019101.456768:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019101.456773:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019101.456781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334578048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019101.456790:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019101.456818:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019101.456820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019101.456823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3368 00000100:00080000:2.0:1734019101.456826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019101.456829:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019101.456834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019101.456843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334578176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019101.456848:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334577792:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019101.456853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019101.456856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019101.456858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019101.456861:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3368 00000100:00080000:2.0:1734019101.456863:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019101.456867:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019101.456871:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019101.456878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334578304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019101.456887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019101.456890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019101.456891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019101.456894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3368 00000100:00080000:2.0:1734019101.456897:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019101.456900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019101.456904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019101.456911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334578432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019101.456914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019101.456919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019101.456922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019101.456923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019101.456926:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3368 00000100:00080000:2.0:1734019101.456929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019101.456933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019101.456934:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019101/real 1734019101] req@ffff8800a78a3b80 x1818247334577792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019101.456937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019101.456944:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334578560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019101.456954:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334577792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019101.456964:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019101.456967:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019101.456972:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334577792:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019101.456979:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334577920:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019101.456996:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019101.457003:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019101/real 1734019101] req@ffff8800a78a1880 x1818247334577920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019101.457014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334577920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019101.457022:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019101.457025:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019101.457028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334577920:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019101.457034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334578048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019101.457047:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019101.457053:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019101/real 1734019101] req@ffff8800a78a2680 x1818247334578048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019101.457063:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334578048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019101.457072:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019101.457074:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019101.457077:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334578048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019101.457100:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334578176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019101.457113:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019101.457119:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019101/real 1734019101] req@ffff8800a78a0a80 x1818247334578176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019101.457130:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334578176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019101.457138:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019101.457140:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019101.457143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334578176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019101.457148:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334578304:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019101.457160:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019101.457166:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019101/real 1734019101] req@ffff8800a78a1f80 x1818247334578304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019101.457176:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334578304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019101.457183:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019101.457185:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019101.457188:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334578304:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019101.457194:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334578432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019101.457206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019101.457212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019101/real 1734019101] req@ffff8800a78a3480 x1818247334578432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019101.457222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334578432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019101.457229:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019101.457231:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019101.457234:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334578432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019101.457239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334578560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019101.457251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019101.457257:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019101/real 1734019101] req@ffff8800a78a0700 x1818247334578560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019101.457267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334578560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019101.457274:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019101.457276:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019101.457279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334578560:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019101.544684:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019102.546702:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019103.548710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019104.550778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019105.552778:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019106.464577:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019106.464587:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019106.464592:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019106.464603:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3373 00000100:00080000:2.0:1734019106.464612:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019106.464620:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019106.464658:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334578688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019106.464692:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019106.464697:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019106.464698:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019106.464703:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3373 00000100:00080000:2.0:1734019106.464706:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019106.464737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019106.464747:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019106.464760:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334578816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019106.464771:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019106.464774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019106.464776:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019106.464780:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3373 00000100:00080000:2.0:1734019106.464783:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019106.464786:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019106.464791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019106.464800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334578944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019106.464811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019106.464814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019106.464816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019106.464819:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3373 00000100:00080000:2.0:1734019106.464821:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019106.464824:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334578688:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019106.464824:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019106.464829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019106.464836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334579072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019106.464845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019106.464847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019106.464849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019106.464852:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3373 00000100:00080000:2.0:1734019106.464854:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019106.464857:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019106.464861:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019106.464869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334579200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019106.464877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019106.464880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019106.464881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019106.464884:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3373 00000100:00080000:2.0:1734019106.464888:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019106.464890:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019106.464892:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019106.464896:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019106.464903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334579328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019106.464911:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019106/real 1734019106] req@ffff8800a78a0700 x1818247334578688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019106.464911:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019106.464914:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019106.464916:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019106.464919:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3373 00000100:00080000:2.0:1734019106.464922:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019106.464925:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019106.464929:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019106.464931:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334578688/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019106.464936:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334579456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019106.464942:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019106.464945:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019106.464950:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334578688:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019106.464958:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334578816:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019106.464975:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019106.464982:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019106/real 1734019106] req@ffff8800a78a3480 x1818247334578816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019106.464993:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334578816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019106.465001:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019106.465003:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019106.465006:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334578816:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019106.465029:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334578944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019106.465044:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019106.465050:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019106/real 1734019106] req@ffff8800a78a1f80 x1818247334578944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019106.465060:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334578944/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019106.465068:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019106.465071:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019106.465074:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334578944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019106.465080:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334579072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019106.465092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019106.465098:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019106/real 1734019106] req@ffff8800a78a0a80 x1818247334579072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019106.465108:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334579072/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019106.465116:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019106.465118:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019106.465121:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334579072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019106.465126:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334579200:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019106.465139:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019106.465145:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019106/real 1734019106] req@ffff8800a78a2680 x1818247334579200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019106.465155:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334579200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019106.465162:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019106.465164:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019106.465167:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334579200:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019106.465173:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334579328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019106.465188:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019106.465194:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019106/real 1734019106] req@ffff8800a78a1880 x1818247334579328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019106.465205:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334579328/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019106.465212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019106.465214:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019106.465217:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334579328:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019106.465222:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334579456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019106.465234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019106.465240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019106/real 1734019106] req@ffff8800a78a3b80 x1818247334579456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019106.465250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334579456/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019106.465257:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019106.465259:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019106.465262:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334579456:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019106.554779:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019107.556782:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019108.558744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019109.560808:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019110.562720:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019111.472604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019111.472616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019111.472658:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019111.472674:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3378 00000100:00080000:2.0:1734019111.472687:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019111.472696:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019111.472748:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334579584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019111.472788:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019111.472792:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019111.472794:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019111.472799:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3378 00000100:00080000:2.0:1734019111.472803:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019111.472807:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019111.472818:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019111.472831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334579712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019111.472841:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019111.472844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019111.472846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019111.472850:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3378 00000100:00080000:2.0:1734019111.472853:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019111.472857:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019111.472862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019111.472872:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334579840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019111.472881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019111.472884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019111.472886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019111.472890:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3378 00000100:00080000:2.0:1734019111.472893:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019111.472896:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019111.472900:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019111.472909:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334579968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019111.472919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019111.472922:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334579584:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019111.472922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019111.472923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019111.472927:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3378 00000100:00080000:2.0:1734019111.472930:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019111.472933:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019111.472938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019111.472946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334580096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019111.472954:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019111.472957:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019111.472958:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019111.472962:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3378 00000100:00080000:2.0:1734019111.472964:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019111.472968:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019111.472972:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019111.472980:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334580224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019111.472990:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019111.472993:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734019111.472995:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019111.472995:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019111.472999:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3378 00000100:00080000:2.0:1734019111.473003:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019111.473006:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019111.473010:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019111.473018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019111/real 1734019111] req@ffff8800a78a3b80 x1818247334579584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019111.473018:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334580352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019111.473046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334579584/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019111.473057:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019111.473061:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019111.473066:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334579584:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019111.473074:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334579712:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019111.473093:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019111.473100:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019111/real 1734019111] req@ffff8800a78a1880 x1818247334579712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019111.473111:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334579712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019111.473119:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019111.473122:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019111.473126:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334579712:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019111.473132:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334579840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019111.473146:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019111.473153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019111/real 1734019111] req@ffff8800a78a2680 x1818247334579840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019111.473163:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334579840/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019111.473170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019111.473174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019111.473177:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334579840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019111.473205:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334579968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019111.473218:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019111.473224:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019111/real 1734019111] req@ffff8800a78a0a80 x1818247334579968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019111.473235:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334579968/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019111.473243:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019111.473245:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019111.473248:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334579968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019111.473254:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334580096:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019111.473267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019111.473273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019111/real 1734019111] req@ffff8800a78a1f80 x1818247334580096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019111.473283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334580096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019111.473291:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019111.473293:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019111.473296:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334580096:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019111.473302:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334580224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019111.473318:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019111.473324:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019111/real 1734019111] req@ffff8800a78a3480 x1818247334580224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019111.473334:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334580224/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019111.473341:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019111.473344:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019111.473347:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334580224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019111.473352:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334580352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019111.473365:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019111.473371:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019111/real 1734019111] req@ffff8800a78a0700 x1818247334580352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019111.473380:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334580352/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019111.473387:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019111.473390:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019111.473393:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334580352:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019111.564779:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019112.566703:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019113.568812:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019114.570773:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019115.572788:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019116.480635:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019116.480646:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019116.480652:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019116.480665:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3383 00000100:00080000:2.0:1734019116.480677:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019116.480687:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019116.480735:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334580480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019116.480772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019116.480776:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019116.480779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019116.480784:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3383 00000100:00080000:2.0:1734019116.480787:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019116.480791:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019116.480803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019116.480816:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334580608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019116.480826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019116.480829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019116.480831:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019116.480836:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3383 00000100:00080000:2.0:1734019116.480839:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019116.480842:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019116.480847:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019116.480856:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334580736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019116.480865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019116.480869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019116.480870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019116.480874:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3383 00000100:00080000:2.0:1734019116.480876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019116.480880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019116.480882:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334580480:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019116.480884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019116.480893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334580864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019116.480903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019116.480906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019116.480908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019116.480911:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3383 00000100:00080000:2.0:1734019116.480914:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019116.480917:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019116.480921:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019116.480929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334580992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019116.480937:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019116.480941:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019116.480942:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019116.480946:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3383 00000100:00080000:2.0:1734019116.480949:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019116.480952:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019116.480956:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019116.480964:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334581120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019116.480969:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019116.480972:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019116.480975:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019116.480977:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019116.480980:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3383 00000100:00080000:2.0:1734019116.480984:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019116.480988:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019116.480992:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019116.480995:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019116/real 1734019116] req@ffff8800a78a0700 x1818247334580480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019116.481000:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334581248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019116.481026:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334580480/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019116.481038:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019116.481041:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019116.481047:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334580480:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019116.481054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334580608:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019116.481073:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019116.481080:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019116/real 1734019116] req@ffff8800a78a3480 x1818247334580608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019116.481091:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334580608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019116.481099:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019116.481101:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019116.481105:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334580608:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019116.481131:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334580736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019116.481146:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019116.481152:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019116/real 1734019116] req@ffff8800a78a1f80 x1818247334580736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019116.481162:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334580736/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019116.481170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019116.481173:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019116.481177:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334580736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019116.481183:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334580864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019116.481195:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019116.481201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019116/real 1734019116] req@ffff8800a78a0a80 x1818247334580864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019116.481211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334580864/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019116.481218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019116.481221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019116.481223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334580864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019116.481229:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334580992:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019116.481242:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019116.481248:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019116/real 1734019116] req@ffff8800a78a2680 x1818247334580992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019116.481258:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334580992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019116.481265:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019116.481267:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019116.481270:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334580992:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019116.481276:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334581120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019116.481293:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019116.481299:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019116/real 1734019116] req@ffff8800a78a1880 x1818247334581120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019116.481310:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334581120/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019116.481317:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019116.481319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019116.481322:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334581120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019116.481327:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334581248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019116.481340:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019116.481346:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019116/real 1734019116] req@ffff8800a78a3b80 x1818247334581248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019116.481355:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334581248/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019116.481362:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019116.481364:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019116.481367:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334581248:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019116.574762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019117.576774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019118.578742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019119.580734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019120.582720:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019121.488584:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019121.488594:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019121.488599:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019121.488609:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3388 00000100:00080000:2.0:1734019121.488618:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019121.488625:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019121.488663:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334581376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019121.488698:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019121.488701:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019121.488703:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019121.488708:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3388 00000100:00080000:2.0:1734019121.488711:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019121.488715:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019121.488724:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019121.488737:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334581504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019121.488746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019121.488750:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019121.488751:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019121.488755:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3388 00000100:00080000:2.0:1734019121.488757:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019121.488761:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019121.488765:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019121.488774:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334581632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019121.488782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019121.488785:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019121.488787:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019121.488790:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3388 00000100:00080000:2.0:1734019121.488793:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019121.488796:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019121.488800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019121.488808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334581760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019121.488816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019121.488819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019121.488821:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019121.488825:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3388 00000100:00080000:2.0:1734019121.488828:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019121.488831:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019121.488835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019121.488843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334581888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019121.488846:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334581376:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019121.488853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019121.488855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019121.488857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019121.488860:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3388 00000100:00080000:2.0:1734019121.488863:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019121.488866:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019121.488870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019121.488878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334582016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019121.488886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019121.488889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019121.488890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019121.488893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3388 00000100:00080000:2.0:1734019121.488895:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019121.488899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019121.488903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019121.488910:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334582144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019121.488943:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019121.488971:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019121/real 1734019121] req@ffff8800a78a3b80 x1818247334581376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019121.489001:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334581376/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019121.489013:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019121.489017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019121.489022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334581376:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019121.489031:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334581504:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019121.489050:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019121.489057:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019121/real 1734019121] req@ffff8800a78a1880 x1818247334581504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019121.489069:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334581504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019121.489076:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019121.489079:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019121.489083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334581504:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019121.489089:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334581632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019121.489105:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019121.489111:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019121/real 1734019121] req@ffff8800a78a2680 x1818247334581632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019121.489121:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334581632/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019121.489128:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019121.489132:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019121.489135:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334581632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019121.489142:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334581760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019121.489156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019121.489162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019121/real 1734019121] req@ffff8800a78a0a80 x1818247334581760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019121.489172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334581760/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019121.489180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019121.489182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019121.489185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334581760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019121.489216:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334581888:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019121.489230:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019121.489236:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019121/real 1734019121] req@ffff8800a78a1f80 x1818247334581888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019121.489247:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334581888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019121.489254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019121.489256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019121.489259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334581888:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019121.489266:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334582016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019121.489283:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019121.489289:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019121/real 1734019121] req@ffff8800a78a3480 x1818247334582016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019121.489300:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334582016/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019121.489307:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019121.489309:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019121.489312:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334582016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019121.489318:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334582144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019121.489331:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019121.489337:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019121/real 1734019121] req@ffff8800a78a0700 x1818247334582144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019121.489346:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334582144/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019121.489353:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019121.489356:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019121.489359:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334582144:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019121.584727:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019122.586706:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019123.588712:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019124.590645:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019125.592738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019126.496600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019126.496611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019126.496616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019126.496626:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3393 00000100:00080000:2.0:1734019126.496635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019126.496643:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019126.496681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334582272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019126.496717:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019126.496721:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019126.496723:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019126.496728:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3393 00000100:00080000:2.0:1734019126.496731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019126.496734:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019126.496744:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019126.496755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334582400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019126.496765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019126.496768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019126.496770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019126.496774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3393 00000100:00080000:2.0:1734019126.496777:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019126.496780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019126.496785:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019126.496793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334582528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019126.496802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019126.496806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019126.496807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019126.496811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3393 00000100:00080000:2.0:1734019126.496813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019126.496816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019126.496820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019126.496828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334582656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019126.496839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019126.496842:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019126.496844:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019126.496847:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3393 00000100:00080000:2.0:1734019126.496850:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019126.496853:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019126.496857:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019126.496862:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334582272:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019126.496865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334582784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019126.496874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019126.496877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019126.496878:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019126.496882:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3393 00000100:00080000:2.0:1734019126.496884:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019126.496887:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019126.496892:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019126.496899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334582912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019126.496907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019126.496910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019126.496911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019126.496915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3393 00000100:00080000:2.0:1734019126.496917:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019126.496920:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019126.496924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019126.496931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334583040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019126.496961:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019126.496988:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019126/real 1734019126] req@ffff8800a78a0700 x1818247334582272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019126.497018:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334582272/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019126.497030:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019126.497034:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019126.497039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334582272:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019126.497048:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334582400:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019126.497066:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019126.497073:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019126/real 1734019126] req@ffff8800a78a3480 x1818247334582400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019126.497085:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334582400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019126.497093:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019126.497096:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019126.497100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334582400:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019126.497106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334582528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019126.497120:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019126.497127:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019126/real 1734019126] req@ffff8800a78a1f80 x1818247334582528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019126.497137:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334582528/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019126.497144:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019126.497148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019126.497151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334582528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019126.497182:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334582656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019126.497196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019126.497203:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019126/real 1734019126] req@ffff8800a78a0a80 x1818247334582656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019126.497214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334582656/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019126.497222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019126.497224:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019126.497227:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334582656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019126.497233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334582784:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019126.497246:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019126.497252:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019126/real 1734019126] req@ffff8800a78a2680 x1818247334582784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019126.497262:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334582784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019126.497269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019126.497272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019126.497275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334582784:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019126.497281:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334582912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019126.497298:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019126.497304:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019126/real 1734019126] req@ffff8800a78a1880 x1818247334582912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019126.497315:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334582912/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019126.497322:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019126.497325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019126.497327:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334582912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019126.497333:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334583040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019126.497347:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019126.497353:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019126/real 1734019126] req@ffff8800a78a3b80 x1818247334583040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019126.497362:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334583040/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019126.497415:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019126.497417:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019126.497421:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334583040:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019126.594712:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019127.596737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019128.598739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019129.600734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019130.602731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019131.504604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019131.504616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019131.504622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019131.504636:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3398 00000100:00080000:2.0:1734019131.504649:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019131.504658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019131.504709:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334583168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019131.504750:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019131.504754:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019131.504756:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019131.504761:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3398 00000100:00080000:2.0:1734019131.504765:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019131.504769:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019131.504780:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019131.504793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334583296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019131.504802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019131.504806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019131.504808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019131.504812:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3398 00000100:00080000:2.0:1734019131.504815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019131.504819:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019131.504823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019131.504833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334583424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019131.504842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019131.504846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019131.504848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019131.504851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3398 00000100:00080000:2.0:1734019131.504854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019131.504857:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019131.504861:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019131.504870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334583552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019131.504879:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019131.504880:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334583168:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019131.504882:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019131.504884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019131.504888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3398 00000100:00080000:2.0:1734019131.504890:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019131.504893:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019131.504897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019131.504906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334583680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019131.504914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019131.504917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019131.504918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019131.504922:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3398 00000100:00080000:2.0:1734019131.504925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019131.504929:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019131.504933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019131.504942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334583808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019131.504949:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019131.504952:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019131.504954:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019131.504957:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3398 00000100:00080000:2.0:1734019131.504960:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019131.504963:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019131.504967:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019131.504970:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019131.504975:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334583936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019131.504996:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019131/real 1734019131] req@ffff8800a78a3b80 x1818247334583168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019131.505027:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334583168/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019131.505038:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019131.505041:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019131.505046:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334583168:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019131.505053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334583296:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019131.505072:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019131.505079:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019131/real 1734019131] req@ffff8800a78a1880 x1818247334583296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019131.505091:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334583296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019131.505099:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019131.505101:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019131.505104:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334583296:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019131.505110:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334583424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019131.505125:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019131.505131:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019131/real 1734019131] req@ffff8800a78a2680 x1818247334583424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019131.505141:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334583424/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019131.505148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019131.505152:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019131.505155:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334583424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019131.505184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334583552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019131.505199:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019131.505205:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019131/real 1734019131] req@ffff8800a78a0a80 x1818247334583552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019131.505216:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334583552/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019131.505223:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019131.505226:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019131.505229:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334583552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019131.505234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334583680:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019131.505247:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019131.505253:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019131/real 1734019131] req@ffff8800a78a1f80 x1818247334583680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019131.505263:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334583680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019131.505270:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019131.505272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019131.505275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334583680:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019131.505313:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334583808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019131.505335:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019131.505343:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019131/real 1734019131] req@ffff8800a78a3480 x1818247334583808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019131.505354:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334583808/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019131.505362:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019131.505364:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019131.505367:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334583808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019131.505373:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334583936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019131.505386:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019131.505392:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019131/real 1734019131] req@ffff8800a78a0700 x1818247334583936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019131.505402:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334583936/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019131.505410:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019131.505412:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019131.505415:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334583936:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019131.604729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019132.606729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019133.608736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019134.610720:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019135.612738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019136.512606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019136.512617:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019136.512621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019136.512631:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3403 00000100:00080000:2.0:1734019136.512640:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019136.512648:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019136.512683:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334584064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019136.512718:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019136.512722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019136.512724:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019136.512728:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3403 00000100:00080000:2.0:1734019136.512731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019136.512735:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019136.512744:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019136.512755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334584192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019136.512765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019136.512768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019136.512769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019136.512773:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3403 00000100:00080000:2.0:1734019136.512776:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019136.512779:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019136.512784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019136.512792:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334584320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019136.512801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019136.512805:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019136.512806:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019136.512810:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3403 00000100:00080000:2.0:1734019136.512813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019136.512816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019136.512820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019136.512828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334584448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019136.512837:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019136.512840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019136.512842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019136.512845:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3403 00000100:00100000:1.0:1734019136.512847:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334584064:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019136.512848:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019136.512851:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019136.512855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019136.512863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334584576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019136.512871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019136.512874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019136.512876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019136.512879:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3403 00000100:00080000:2.0:1734019136.512881:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019136.512884:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019136.512889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019136.512896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334584704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019136.512905:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019136.512909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019136.512910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019136.512913:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3403 00000100:00080000:2.0:1734019136.512916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019136.512919:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019136.512923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019136.512930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334584832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019136.512942:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019136.512970:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019136/real 1734019136] req@ffff8800a78a0700 x1818247334584064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019136.513000:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334584064/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019136.513013:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019136.513017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019136.513022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334584064:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019136.513030:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334584192:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019136.513050:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019136.513057:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019136/real 1734019136] req@ffff8800a78a3480 x1818247334584192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019136.513068:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334584192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019136.513076:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019136.513079:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019136.513083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334584192:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019136.513089:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334584320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019136.513104:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019136.513110:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019136/real 1734019136] req@ffff8800a78a1f80 x1818247334584320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019136.513120:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334584320/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019136.513128:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019136.513131:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019136.513134:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334584320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019136.513166:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334584448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019136.513181:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019136.513187:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019136/real 1734019136] req@ffff8800a78a0a80 x1818247334584448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019136.513233:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334584448/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019136.513242:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019136.513245:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019136.513249:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334584448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019136.513255:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334584576:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019136.513272:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019136.513278:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019136/real 1734019136] req@ffff8800a78a2680 x1818247334584576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019136.513289:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334584576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019136.513298:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019136.513300:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019136.513303:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334584576:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019136.513309:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334584704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019136.513326:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019136.513332:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019136/real 1734019136] req@ffff8800a78a1880 x1818247334584704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019136.513343:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334584704/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019136.513350:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019136.513353:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019136.513356:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334584704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019136.513361:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334584832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019136.513375:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019136.513381:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019136/real 1734019136] req@ffff8800a78a3b80 x1818247334584832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019136.513391:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334584832/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019136.513398:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019136.513400:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019136.513403:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334584832:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019136.614632:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019137.616722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019138.618697:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019139.620732:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019140.622722:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019141.520592:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019141.520602:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019141.520606:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019141.520617:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3408 00000100:00080000:2.0:1734019141.520626:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019141.520635:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019141.520674:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334584960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019141.520709:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019141.520713:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019141.520715:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019141.520720:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3408 00000100:00080000:2.0:1734019141.520723:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019141.520727:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019141.520736:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019141.520748:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334585088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019141.520758:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019141.520761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019141.520763:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019141.520766:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3408 00000100:00080000:2.0:1734019141.520769:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019141.520773:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019141.520777:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019141.520786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334585216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019141.520795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019141.520799:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019141.520800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019141.520803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3408 00000100:00080000:2.0:1734019141.520806:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019141.520809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019141.520813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019141.520820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334585344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019141.520831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019141.520834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019141.520835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019141.520839:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3408 00000100:00080000:2.0:1734019141.520841:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019141.520845:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019141.520849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019141.520855:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334584960:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019141.520856:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334585472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019141.520864:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019141.520867:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019141.520869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019141.520872:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3408 00000100:00080000:2.0:1734019141.520874:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019141.520877:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019141.520882:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019141.520890:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334585600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019141.520897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019141.520900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019141.520902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019141.520905:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3408 00000100:00080000:2.0:1734019141.520908:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019141.520911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019141.520915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019141.520922:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334585728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019141.520950:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019141.520977:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019141/real 1734019141] req@ffff8800a78a3b80 x1818247334584960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019141.521008:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334584960/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019141.521020:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019141.521024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019141.521029:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334584960:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019141.521038:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334585088:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019141.521057:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019141.521064:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019141/real 1734019141] req@ffff8800a78a1880 x1818247334585088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019141.521075:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334585088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019141.521083:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019141.521085:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019141.521089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334585088:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019141.521095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334585216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019141.521157:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019141.521167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019141/real 1734019141] req@ffff8800a78a2680 x1818247334585216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019141.521178:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334585216/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019141.521186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019141.521189:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019141.521193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334585216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019141.521225:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334585344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019141.521244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019141.521250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019141/real 1734019141] req@ffff8800a78a0a80 x1818247334585344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019141.521261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334585344/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019141.521269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019141.521272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019141.521275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334585344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019141.521281:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334585472:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019141.521294:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019141.521300:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019141/real 1734019141] req@ffff8800a78a1f80 x1818247334585472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019141.521310:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334585472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019141.521318:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019141.521320:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019141.521324:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334585472:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019141.521330:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334585600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019141.521346:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019141.521353:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019141/real 1734019141] req@ffff8800a78a3480 x1818247334585600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019141.521363:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334585600/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019141.521370:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019141.521373:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019141.521376:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334585600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019141.521381:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334585728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019141.521395:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019141.521401:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019141/real 1734019141] req@ffff8800a78a0700 x1818247334585728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019141.521411:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334585728/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019141.521418:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019141.521420:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019141.521423:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334585728:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019141.624719:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019142.626740:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019143.628745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019144.630703:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019145.632731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019146.528588:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019146.528599:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019146.528603:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019146.528613:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3413 00000100:00080000:2.0:1734019146.528622:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019146.528630:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019146.528667:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334585856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019146.528701:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019146.528705:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019146.528707:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019146.528712:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3413 00000100:00080000:2.0:1734019146.528715:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019146.528718:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019146.528728:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019146.528739:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334585984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019146.528749:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019146.528752:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019146.528754:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019146.528758:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3413 00000100:00080000:2.0:1734019146.528761:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019146.528764:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019146.528768:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019146.528777:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334586112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019146.528785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019146.528788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019146.528789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019146.528793:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3413 00000100:00080000:2.0:1734019146.528796:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019146.528799:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019146.528803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019146.528810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334586240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019146.528818:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019146.528821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019146.528823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019146.528827:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3413 00000100:00080000:2.0:1734019146.528830:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019146.528833:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019146.528837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019146.528844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334586368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019146.528853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019146.528856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1734019146.528858:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334585856:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019146.528858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019146.528861:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3413 00000100:00080000:2.0:1734019146.528864:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019146.528867:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019146.528871:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019146.528880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334586496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019146.528888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019146.528891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019146.528892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019146.528896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3413 00000100:00080000:2.0:1734019146.528898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019146.528901:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019146.528905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019146.528912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334586624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019146.528956:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019146.528983:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019146/real 1734019146] req@ffff8800a78a0700 x1818247334585856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019146.529014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334585856/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019146.529070:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019146.529076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019146.529082:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334585856:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019146.529092:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334585984:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019146.529116:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019146.529125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019146/real 1734019146] req@ffff8800a78a3480 x1818247334585984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019146.529137:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334585984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019146.529145:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019146.529148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019146.529151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334585984:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019146.529158:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334586112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019146.529173:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019146.529179:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019146/real 1734019146] req@ffff8800a78a1f80 x1818247334586112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019146.529189:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334586112/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019146.529198:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019146.529201:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019146.529205:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334586112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019146.529211:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334586240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019146.529225:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019146.529231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019146/real 1734019146] req@ffff8800a78a0a80 x1818247334586240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019146.529241:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334586240/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019146.529249:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019146.529251:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019146.529254:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334586240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019146.529286:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334586368:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019146.529301:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019146.529307:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019146/real 1734019146] req@ffff8800a78a2680 x1818247334586368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019146.529317:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334586368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019146.529326:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019146.529328:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019146.529332:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334586368:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019146.529338:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334586496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019146.529354:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019146.529361:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019146/real 1734019146] req@ffff8800a78a1880 x1818247334586496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019146.529372:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334586496/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019146.529379:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019146.529381:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019146.529384:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334586496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019146.529390:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334586624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019146.529403:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019146.529409:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019146/real 1734019146] req@ffff8800a78a3b80 x1818247334586624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019146.529419:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334586624/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019146.529426:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019146.529429:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019146.529432:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334586624:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019146.634732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019147.636738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019148.638761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019149.640724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019150.642736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019151.536590:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019151.536601:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019151.536606:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019151.536616:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3418 00000100:00080000:2.0:1734019151.536625:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019151.536632:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019151.536669:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334586752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019151.536706:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019151.536710:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019151.536712:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019151.536716:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3418 00000100:00080000:2.0:1734019151.536719:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019151.536723:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019151.536732:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019151.536743:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334586880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019151.536753:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019151.536756:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019151.536758:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019151.536762:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3418 00000100:00080000:2.0:1734019151.536765:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019151.536768:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019151.536773:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019151.536781:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334587008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019151.536790:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019151.536793:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019151.536794:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019151.536798:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3418 00000100:00080000:2.0:1734019151.536801:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019151.536805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019151.536808:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019151.536816:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334587136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019151.536824:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019151.536827:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019151.536829:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019151.536832:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3418 00000100:00080000:2.0:1734019151.536834:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019151.536838:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019151.536842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019151.536849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334587264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019151.536859:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019151.536862:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1734019151.536863:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334586752:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019151.536864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019151.536867:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3418 00000100:00080000:2.0:1734019151.536870:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019151.536873:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019151.536877:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019151.536885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334587392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019151.536893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019151.536896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019151.536897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019151.536901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3418 00000100:00080000:2.0:1734019151.536903:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019151.536906:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019151.536910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019151.536917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334587520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019151.537005:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019151.537034:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019151/real 1734019151] req@ffff8800a78a3b80 x1818247334586752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019151.537064:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334586752/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019151.537076:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019151.537080:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019151.537085:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334586752:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019151.537094:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334586880:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019151.537115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019151.537122:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019151/real 1734019151] req@ffff8800a78a1880 x1818247334586880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019151.537134:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334586880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019151.537142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019151.537144:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019151.537148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334586880:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019151.537155:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334587008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019151.537170:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019151.537176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019151/real 1734019151] req@ffff8800a78a2680 x1818247334587008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019151.537186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334587008/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019151.537194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019151.537197:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019151.537200:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334587008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019151.537207:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334587136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019151.537221:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019151.537227:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019151/real 1734019151] req@ffff8800a78a0a80 x1818247334587136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019151.537237:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334587136/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019151.537245:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019151.537247:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019151.537250:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334587136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019151.537282:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334587264:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019151.537296:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019151.537302:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019151/real 1734019151] req@ffff8800a78a1f80 x1818247334587264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019151.537313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334587264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019151.537320:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019151.537322:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019151.537325:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334587264:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019151.537331:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334587392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019151.537348:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019151.537354:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019151/real 1734019151] req@ffff8800a78a3480 x1818247334587392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019151.537365:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334587392/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019151.537372:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019151.537374:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019151.537377:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334587392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019151.537383:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334587520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019151.537396:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019151.537402:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019151/real 1734019151] req@ffff8800a78a0700 x1818247334587520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019151.537412:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334587520/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019151.537419:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019151.537421:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019151.537424:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334587520:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019151.644723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019152.646700:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019153.648734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019154.650724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019155.652747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019156.544722:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019156.544733:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019156.544737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019156.544748:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3423 00000100:00080000:0.0:1734019156.544758:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019156.544765:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019156.544802:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334587648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019156.544826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019156.544830:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019156.544832:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019156.544836:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3423 00000100:00080000:0.0:1734019156.544839:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019156.544843:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734019156.544850:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334587648:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734019156.544852:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019156.544902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334587776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019156.544914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019156.544917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019156.544919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019156.544923:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3423 00000100:00080000:0.0:1734019156.544928:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019156.544932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019156.544938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1734019156.544945:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019156.544949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334587904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019156.544959:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019156.544962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019156.544963:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019156.544966:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019156/real 1734019156] req@ffff8800a934ce00 x1818247334587648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019156.544967:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3423 00000100:00080000:0.0:1734019156.544970:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019156.544973:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019156.544977:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019156.544986:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334588032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019156.544986:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334587648/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019156.544995:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019156.544997:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019156.544998:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019156.544999:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019156.545000:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734019156.545002:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3423 00000100:00080000:0.0:1734019156.545005:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019156.545005:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334587648:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734019156.545008:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019156.545012:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019156.545020:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334588160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019156.545028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334587776:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734019156.545036:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019156.545039:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019156.545040:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019156.545044:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3423 00000800:00000100:2.0:1734019156.545046:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734019156.545047:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019156.545050:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019156.545054:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019156.545054:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019156/real 1734019156] req@ffff8800a934e680 x1818247334587776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019156.545062:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334588288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019156.545065:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334587776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019156.545070:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019156.545073:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019156.545073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019156.545074:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019156.545075:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00080000:0.0:1734019156.545078:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3423 00000100:00100000:2.0:1734019156.545079:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334587776:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734019156.545080:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019156.545084:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734019156.545085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334587904:192.168.201.116@tcp:8:kworker.0 00000008:00100000:0.0:1734019156.545088:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019156.545096:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334588416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019156.545099:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019156.545105:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019156/real 1734019156] req@ffff8800a934d500 x1818247334587904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019156.545116:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334587904/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019156.545128:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019156.545131:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019156.545134:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334587904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019156.545175:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334588032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019156.545193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019156.545201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019156/real 1734019156] req@ffff8800a934fb80 x1818247334588032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019156.545211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334588032/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019156.545219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019156.545222:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019156.545225:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334588032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019156.545240:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334588160:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019156.545253:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019156.545259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019156/real 1734019156] req@ffff8800a934f100 x1818247334588160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019156.545269:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334588160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019156.545277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019156.545279:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019156.545283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334588160:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019156.545289:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334588288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019156.545301:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019156.545307:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019156/real 1734019156] req@ffff8800a934e300 x1818247334588288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019156.545317:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334588288/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019156.545324:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019156.545327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019156.545330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334588288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019156.545335:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334588416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019156.545347:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019156.545353:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019156/real 1734019156] req@ffff8800a934dc00 x1818247334588416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019156.545362:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334588416/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019156.545369:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019156.545372:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019156.545375:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334588416:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019156.654728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019157.656735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019158.658745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019159.660715:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019160.662706:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019161.552619:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019161.552630:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019161.552636:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019161.552650:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3428 00000100:00080000:0.0:1734019161.552662:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019161.552673:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019161.552721:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334588544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019161.552760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019161.552765:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019161.552767:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019161.552808:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3428 00000100:00080000:0.0:1734019161.552812:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019161.552816:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019161.552829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019161.552843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334588672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019161.552854:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019161.552858:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019161.552860:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019161.552864:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3428 00000100:00080000:0.0:1734019161.552867:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019161.552871:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019161.552875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019161.552886:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334588800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019161.552893:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334588544:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734019161.552896:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019161.552899:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019161.552902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019161.552906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3428 00000100:00080000:0.0:1734019161.552909:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019161.552912:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019161.552917:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019161.552925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334588928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019161.552934:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019161.552937:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019161.552938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019161.552942:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3428 00000100:00080000:0.0:1734019161.552944:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019161.552948:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019161.552952:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019161.552960:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334589056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019161.552968:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019161.552971:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019161.552972:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019161.552976:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3428 00000100:00080000:0.0:1734019161.552979:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019161.552983:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019161.552987:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019161.552997:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334588672:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019161.552998:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334589184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019161.553008:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019161.553011:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019161.553013:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019161.553016:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3428 00000100:00080000:0.0:1734019161.553020:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019161.553023:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019161.553027:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019161.553028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334588800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019161.553036:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334589312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019161.553047:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334588928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019161.553063:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334589056:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019161.553079:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334589184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019161.553096:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334589312:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019161.664705:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019162.666706:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019163.668708:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019164.558795:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019164.558858:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019164.558886:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019164.558937:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019164.558946:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019164.558955:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019164.558963:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019164.558971:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019164.558978:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:2.0:1734019164.559081:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019161/real 1734019164] req@ffff8800a934c380 x1818247334589312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019164.559127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334589312/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019164.559146:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019164.559153:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019164.559161:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334589312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019164.559185:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019161/real 1734019164] req@ffff8800a934c700 x1818247334588800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019164.559207:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334588800/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019164.559220:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019164.559225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019164.559230:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334588800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019164.559241:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019161/real 1734019164] req@ffff8800a934ed80 x1818247334588928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019164.559260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334588928/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019164.559273:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019164.559278:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019164.559283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334588928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019164.559293:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019161/real 1734019164] req@ffff8800a934ca80 x1818247334589056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019164.559313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334589056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019164.559325:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019164.559330:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019164.559335:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334589056:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019164.559346:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019161/real 1734019164] req@ffff8800a934ea00 x1818247334589184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019164.559364:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334589184/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019164.559389:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019164.559393:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019164.559398:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334589184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019164.559409:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019161/real 1734019164] req@ffff8800a934c000 x1818247334588544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019164.559429:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334588544/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019164.559440:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019164.559444:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019164.559449:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334588544:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019164.559460:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019161/real 1734019164] req@ffff8800a934d180 x1818247334588672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019164.559549:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334588672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019164.559562:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019164.559566:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019164.559571:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334588672:192.168.201.116@tcp:38:kworker.0 00010000:00010000:2.0:1734019164.670695:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019165.672761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019166.560582:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019166.560594:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019166.560599:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019166.560613:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3433 00000100:00080000:0.0:1734019166.560626:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019166.560635:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019166.560684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334589440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019166.560746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019166.560750:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019166.560752:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019166.560758:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3433 00000100:00080000:0.0:1734019166.560761:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019166.560765:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019166.560778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019166.560791:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334589568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019166.560801:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019166.560805:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019166.560807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019166.560811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3433 00000100:00080000:0.0:1734019166.560815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019166.560818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019166.560823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019166.560835:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334589696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019166.560845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019166.560848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019166.560850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019166.560853:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3433 00000100:00080000:0.0:1734019166.560856:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019166.560859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019166.560864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019166.560866:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334589440:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734019166.560873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334589824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019166.560882:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019166.560885:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019166.560887:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019166.560891:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3433 00000100:00080000:0.0:1734019166.560894:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019166.560897:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019166.560903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019166.560912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334589952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019166.560920:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019166.560923:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019166.560924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019166.560928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3433 00000100:00080000:0.0:1734019166.560931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019166.560934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019166.560938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019166.560939:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019166.560947:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334590080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019166.560955:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019166.560958:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019166.560959:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019166.560963:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019166/real 1734019166] req@ffff8800a934d880 x1818247334589440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019166.560964:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3433 00000100:00080000:0.0:1734019166.560966:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019166.560970:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019166.560974:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019166.560981:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334590208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019166.560983:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334589440/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019166.560995:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019166.560998:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019166.561004:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334589440:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019166.561012:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334589568:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019166.561030:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019166.561037:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019166/real 1734019166] req@ffff8800a934df80 x1818247334589568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019166.561049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334589568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019166.561056:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019166.561059:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019166.561063:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334589568:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019166.561087:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334589696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019166.561101:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019166.561107:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019166/real 1734019166] req@ffff8800a934d180 x1818247334589696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019166.561118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334589696/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019166.561133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019166.561135:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019166.561139:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334589696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019166.561145:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334589824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019166.561158:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019166.561164:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019166/real 1734019166] req@ffff8800a934c000 x1818247334589824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019166.561174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334589824/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019166.561181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019166.561184:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019166.561187:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334589824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019166.561193:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334589952:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019166.561206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019166.561212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019166/real 1734019166] req@ffff8800a934ea00 x1818247334589952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019166.561222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334589952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019166.561229:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019166.561232:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019166.561235:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334589952:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019166.561242:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334590080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019166.561255:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019166.561261:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019166/real 1734019166] req@ffff8800a934ca80 x1818247334590080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019166.561271:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334590080/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019166.561278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019166.561280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019166.561283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334590080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019166.561289:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334590208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019166.561302:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019166.561308:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019166/real 1734019166] req@ffff8800a934ed80 x1818247334590208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019166.561318:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334590208/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019166.561325:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019166.561327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019166.561330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334590208:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019166.674752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019167.676740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019168.678742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019169.680724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019170.682784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019171.568638:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019171.568649:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019171.568653:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019171.568664:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3438 00000100:00080000:0.0:1734019171.568673:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019171.568680:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019171.568716:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334590336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019171.568749:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019171.568753:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019171.568755:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019171.568759:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3438 00000100:00080000:0.0:1734019171.568762:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019171.568766:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019171.568776:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019171.568787:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334590464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019171.568796:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019171.568799:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019171.568801:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019171.568805:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3438 00000100:00080000:0.0:1734019171.568807:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019171.568811:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019171.568816:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019171.568824:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334590592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019171.568833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019171.568836:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019171.568837:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019171.568840:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3438 00000100:00080000:0.0:1734019171.568843:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019171.568846:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019171.568850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019171.568858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334590720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019171.568865:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334590336:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734019171.568867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019171.568870:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019171.568871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019171.568875:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3438 00000100:00080000:0.0:1734019171.568877:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019171.568880:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019171.568884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019171.568893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334590848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019171.568901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019171.568904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019171.568906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019171.568910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3438 00000100:00080000:0.0:1734019171.568913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019171.568916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019171.568920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019171.568927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334590976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019171.568931:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734019171.568935:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019171.568938:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019171.568940:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019171.568943:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3438 00000100:00080000:0.0:1734019171.568945:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019171.568948:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019171.568952:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019171.568954:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019171/real 1734019171] req@ffff8800a934c700 x1818247334590336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019171.568960:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334591104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019171.568974:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334590336/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019171.568985:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019171.568988:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019171.568993:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334590336:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019171.569000:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334590464:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019171.569017:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019171.569024:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019171/real 1734019171] req@ffff8800a934c380 x1818247334590464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019171.569035:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334590464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019171.569043:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019171.569046:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019171.569049:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334590464:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019171.569054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334590592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019171.569067:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019171.569073:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019171/real 1734019171] req@ffff8800a934dc00 x1818247334590592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019171.569085:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334590592/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019171.569093:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019171.569095:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019171.569098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334590592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019171.569124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334590720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019171.569137:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019171.569143:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019171/real 1734019171] req@ffff8800a934e300 x1818247334590720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019171.569153:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334590720/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019171.569161:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019171.569163:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019171.569166:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334590720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019171.569172:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334590848:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019171.569184:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019171.569190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019171/real 1734019171] req@ffff8800a934f100 x1818247334590848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019171.569200:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334590848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019171.569208:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019171.569210:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019171.569213:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334590848:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019171.569220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334590976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019171.569232:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019171.569238:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019171/real 1734019171] req@ffff8800a934fb80 x1818247334590976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019171.569248:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334590976/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019171.569255:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019171.569257:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019171.569260:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334590976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019171.569265:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334591104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019171.569277:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019171.569283:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019171/real 1734019171] req@ffff8800a934d500 x1818247334591104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019171.569293:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334591104/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019171.569300:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019171.569302:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019171.569305:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334591104:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019171.684747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019172.686795:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019173.688761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019174.690753:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019175.692735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019176.576611:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019176.576621:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019176.576626:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019176.576636:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3443 00000100:00080000:0.0:1734019176.576645:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019176.576653:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019176.576688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334591232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019176.576728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019176.576732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019176.576734:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019176.576738:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3443 00000100:00080000:0.0:1734019176.576742:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019176.576746:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019176.576755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019176.576766:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334591360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019176.576775:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019176.576778:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019176.576780:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019176.576784:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3443 00000100:00080000:0.0:1734019176.576787:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019176.576790:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019176.576795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019176.576805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334591488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019176.576813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019176.576816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019176.576818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019176.576821:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3443 00000100:00080000:0.0:1734019176.576824:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019176.576828:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019176.576832:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019176.576835:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334591232:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734019176.576840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334591616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019176.576849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019176.576852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019176.576854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019176.576857:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3443 00000100:00080000:0.0:1734019176.576859:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019176.576862:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019176.576866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019176.576873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334591744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019176.576881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019176.576884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019176.576885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019176.576889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3443 00000100:00080000:0.0:1734019176.576891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019176.576894:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019176.576898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019176.576903:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019176.576906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334591872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019176.576914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019176.576918:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019176.576919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019176.576923:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3443 00000100:00100000:2.0:1734019176.576925:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019176/real 1734019176] req@ffff8800a934e680 x1818247334591232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019176.576926:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019176.576929:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019176.576933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019176.576940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334592000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019176.576944:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334591232/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019176.576954:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019176.576957:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019176.576962:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334591232:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019176.576969:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334591360:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019176.576986:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019176.576993:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019176/real 1734019176] req@ffff8800a934ce00 x1818247334591360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019176.577004:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334591360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019176.577012:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019176.577014:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019176.577018:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334591360:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019176.577023:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334591488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019176.577036:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019176.577044:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019176/real 1734019176] req@ffff8800a934d500 x1818247334591488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019176.577054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334591488/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019176.577062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019176.577064:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019176.577067:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334591488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019176.577091:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334591616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019176.577104:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019176.577110:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019176/real 1734019176] req@ffff8800a934fb80 x1818247334591616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019176.577120:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334591616/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019176.577127:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019176.577130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019176.577133:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334591616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019176.577139:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334591744:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019176.577151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019176.577157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019176/real 1734019176] req@ffff8800a934f100 x1818247334591744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019176.577166:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334591744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019176.577181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019176.577183:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019176.577187:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334591744:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019176.577192:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334591872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019176.577204:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019176.577210:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019176/real 1734019176] req@ffff8800a934e300 x1818247334591872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019176.577220:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334591872/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019176.577227:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019176.577229:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019176.577232:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334591872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019176.577238:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334592000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019176.577249:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019176.577255:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019176/real 1734019176] req@ffff8800a934dc00 x1818247334592000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019176.577265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334592000/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019176.577272:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019176.577274:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019176.577277:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334592000:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019176.694747:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019177.696722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019178.698744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019179.700740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019180.702745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019181.584604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019181.584613:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019181.584617:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019181.584627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3448 00000100:00080000:0.0:1734019181.584636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019181.584644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019181.584678:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334592128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019181.584713:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019181.584718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019181.584720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019181.584724:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3448 00000100:00080000:0.0:1734019181.584727:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019181.584731:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019181.584740:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019181.584750:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334592256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019181.584761:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019181.584764:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019181.584766:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019181.584769:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3448 00000100:00080000:0.0:1734019181.584772:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019181.584775:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019181.584780:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019181.584788:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334592384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019181.584797:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019181.584800:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019181.584801:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019181.584804:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3448 00000100:00080000:0.0:1734019181.584807:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019181.584810:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019181.584814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019181.584821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334592512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019181.584831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019181.584834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019181.584835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019181.584839:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3448 00000100:00080000:0.0:1734019181.584843:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019181.584845:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334592128:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734019181.584846:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019181.584851:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019181.584858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334592640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019181.584866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019181.584869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019181.584870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019181.584873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3448 00000100:00080000:0.0:1734019181.584876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019181.584879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019181.584883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019181.584892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334592768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019181.584900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019181.584903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019181.584904:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019181.584908:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3448 00000100:00080000:0.0:1734019181.584910:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019181.584913:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019181.584917:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019181.584925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334592896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019181.584941:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019181.584971:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019181/real 1734019181] req@ffff8800a934c380 x1818247334592128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019181.584992:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334592128/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019181.585005:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019181.585009:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019181.585014:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334592128:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019181.585022:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334592256:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019181.585041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019181.585048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019181/real 1734019181] req@ffff8800a934c700 x1818247334592256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019181.585060:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334592256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019181.585068:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019181.585071:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019181.585074:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334592256:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019181.585085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334592384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019181.585100:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019181.585107:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019181/real 1734019181] req@ffff8800a934ed80 x1818247334592384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019181.585118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334592384/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019181.585126:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019181.585128:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019181.585131:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334592384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019181.585162:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334592512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019181.585176:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019181.585182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019181/real 1734019181] req@ffff8800a934ca80 x1818247334592512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019181.585192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334592512/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019181.585200:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019181.585202:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019181.585206:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334592512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019181.585213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334592640:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019181.585226:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019181.585232:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019181/real 1734019181] req@ffff8800a934ea00 x1818247334592640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019181.585243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334592640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019181.585251:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019181.585253:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019181.585256:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334592640:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019181.585262:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334592768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019181.585275:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019181.585281:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019181/real 1734019181] req@ffff8800a934c000 x1818247334592768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019181.585290:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334592768/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019181.585298:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019181.585300:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019181.585303:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334592768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019181.585309:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334592896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019181.585322:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019181.585328:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019181/real 1734019181] req@ffff8800a934d180 x1818247334592896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019181.585338:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334592896/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019181.585345:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019181.585349:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019181.585352:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334592896:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019181.704747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019182.706713:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019183.708723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019184.710740:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019185.712739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019186.592598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019186.592608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019186.592613:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019186.592623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3453 00000100:00080000:0.0:1734019186.592632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019186.592640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019186.592675:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334593024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019186.592708:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019186.592712:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019186.592714:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019186.592719:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3453 00000100:00080000:0.0:1734019186.592722:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019186.592726:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019186.592735:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019186.592746:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334593152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019186.592755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019186.592759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019186.592760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019186.592764:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3453 00000100:00080000:0.0:1734019186.592767:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019186.592770:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019186.592776:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019186.592786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334593280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019186.592795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019186.592798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019186.592800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019186.592803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3453 00000100:00080000:0.0:1734019186.592805:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019186.592809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734019186.592812:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334593024:192.168.201.116@tcp:250:kworker.0 00000008:00100000:0.0:1734019186.592813:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019186.592821:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334593408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019186.592830:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019186.592833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019186.592835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019186.592838:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3453 00000100:00080000:0.0:1734019186.592840:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019186.592843:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019186.592847:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019186.592855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334593536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019186.592863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019186.592866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019186.592868:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019186.592871:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3453 00000100:00080000:0.0:1734019186.592874:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734019186.592876:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734019186.592878:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019186.592882:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019186.592890:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334593664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019186.592898:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019186.592898:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019186/real 1734019186] req@ffff8800a934df80 x1818247334593024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019186.592901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019186.592902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019186.592905:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3453 00000100:00080000:0.0:1734019186.592908:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019186.592911:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019186.592915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019186.592917:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334593024/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019186.592922:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334593792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019186.592928:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019186.592931:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019186.592936:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334593024:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019186.592944:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334593152:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019186.592960:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019186.592967:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019186/real 1734019186] req@ffff8800a934d880 x1818247334593152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019186.592978:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334593152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019186.592986:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019186.592989:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019186.592993:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334593152:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019186.592998:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334593280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019186.593012:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019186.593018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019186/real 1734019186] req@ffff8800a934d180 x1818247334593280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019186.593029:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334593280/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019186.593036:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019186.593038:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019186.593042:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334593280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019186.593066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334593408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019186.593079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019186.593085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019186/real 1734019186] req@ffff8800a934c000 x1818247334593408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019186.593095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334593408/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019186.593103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019186.593105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019186.593108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334593408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019186.593113:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334593536:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019186.593126:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019186.593132:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019186/real 1734019186] req@ffff8800a934ea00 x1818247334593536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019186.593143:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334593536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019186.593150:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019186.593152:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019186.593155:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334593536:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019186.593160:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334593664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019186.593172:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019186.593178:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019186/real 1734019186] req@ffff8800a934ca80 x1818247334593664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019186.593188:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334593664/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019186.593195:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019186.593197:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019186.593200:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334593664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019186.593205:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334593792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019186.593217:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019186.593223:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019186/real 1734019186] req@ffff8800a934ed80 x1818247334593792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019186.593233:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334593792/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019186.593247:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019186.593250:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019186.593253:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334593792:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019186.714656:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019187.716745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019188.718719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019189.720735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019190.722729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019191.600603:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019191.600612:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019191.600616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019191.600627:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3458 00000100:00080000:0.0:1734019191.600636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019191.600643:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019191.600678:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334593920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019191.600719:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019191.600724:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019191.600726:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019191.600730:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3458 00000100:00080000:0.0:1734019191.600734:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019191.600738:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019191.600747:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019191.600758:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334594048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019191.600767:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019191.600770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019191.600772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019191.600776:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3458 00000100:00080000:0.0:1734019191.600778:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019191.600782:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019191.600786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019191.600794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334594176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019191.600803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019191.600806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019191.600808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019191.600811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3458 00000100:00080000:0.0:1734019191.600814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019191.600818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019191.600822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019191.600829:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334593920:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734019191.600830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334594304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019191.600838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019191.600841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019191.600843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019191.600846:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3458 00000100:00080000:0.0:1734019191.600848:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019191.600851:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019191.600855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019191.600865:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334594432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019191.600873:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019191.600875:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019191.600877:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019191.600880:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3458 00000100:00080000:0.0:1734019191.600883:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019191.600886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019191.600890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019191.600892:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019191.600898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334594560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019191.600906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019191.600909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019191.600911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019191.600914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3458 00000100:00100000:2.0:1734019191.600914:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019191/real 1734019191] req@ffff8800a934c700 x1818247334593920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019191.600917:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019191.600921:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019191.600924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019191.600932:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334594688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019191.600933:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334593920/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019191.600944:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019191.600947:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019191.600951:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334593920:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019191.600959:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334594048:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019191.600976:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019191.600982:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019191/real 1734019191] req@ffff8800a934c380 x1818247334594048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019191.600994:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334594048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019191.601010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019191.601012:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019191.601016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334594048:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019191.601021:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334594176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019191.601034:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019191.601040:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019191/real 1734019191] req@ffff8800a934dc00 x1818247334594176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019191.601050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334594176/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019191.601057:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019191.601060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019191.601063:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334594176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019191.601087:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334594304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019191.601100:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019191.601107:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019191/real 1734019191] req@ffff8800a934e300 x1818247334594304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019191.601117:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334594304/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019191.601125:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019191.601127:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019191.601130:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334594304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019191.601136:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334594432:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019191.601149:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019191.601155:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019191/real 1734019191] req@ffff8800a934f100 x1818247334594432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019191.601165:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334594432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019191.601173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019191.601175:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019191.601178:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334594432:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019191.601184:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334594560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019191.601195:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019191.601201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019191/real 1734019191] req@ffff8800a934fb80 x1818247334594560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019191.601211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334594560/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019191.601218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019191.601220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019191.601223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334594560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019191.601228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334594688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019191.601241:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019191.601246:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019191/real 1734019191] req@ffff8800a934d500 x1818247334594688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019191.601257:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334594688/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019191.601314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019191.601317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019191.601321:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334594688:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019191.724738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019192.726725:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019193.728724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019194.730731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019195.732707:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019196.608617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019196.608625:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019196.608630:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019196.608641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3463 00000100:00080000:0.0:1734019196.608650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019196.608658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019196.608691:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334594816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019196.608723:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019196.608728:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019196.608730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019196.608734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3463 00000100:00080000:0.0:1734019196.608737:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019196.608741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019196.608750:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019196.608761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334594944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019196.608772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019196.608775:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019196.608777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019196.608780:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3463 00000100:00080000:0.0:1734019196.608783:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019196.608786:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019196.608790:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019196.608800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334595072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019196.608808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019196.608811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019196.608813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019196.608816:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3463 00000100:00080000:0.0:1734019196.608819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019196.608822:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019196.608826:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019196.608833:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334594816:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734019196.608834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334595200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019196.608843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019196.608846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019196.608847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019196.608850:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3463 00000100:00080000:0.0:1734019196.608853:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019196.608856:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019196.608861:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019196.608868:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334595328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019196.608876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019196.608879:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019196.608880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019196.608884:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3463 00000100:00080000:0.0:1734019196.608886:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019196.608889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019196.608893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019196.608898:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019196.608901:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334595456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019196.608909:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019196.608912:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019196.608913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019196.608916:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3463 00000100:00080000:0.0:1734019196.608919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019196.608920:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019196/real 1734019196] req@ffff8800a934ce00 x1818247334594816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019196.608922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019196.608926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019196.608933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334595584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019196.608939:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334594816/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019196.608949:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019196.608952:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019196.608957:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334594816:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019196.608964:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334594944:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019196.608981:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019196.608988:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019196/real 1734019196] req@ffff8800a934e680 x1818247334594944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019196.609007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334594944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019196.609015:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019196.609017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019196.609021:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334594944:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019196.609027:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334595072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019196.609040:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019196.609046:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019196/real 1734019196] req@ffff8800a934d500 x1818247334595072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019196.609056:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334595072/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019196.609063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019196.609065:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019196.609068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334595072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019196.609093:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334595200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019196.609106:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019196.609113:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019196/real 1734019196] req@ffff8800a934fb80 x1818247334595200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019196.609123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334595200/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019196.609130:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019196.609133:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019196.609136:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334595200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019196.609142:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334595328:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019196.609155:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019196.609161:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019196/real 1734019196] req@ffff8800a934f100 x1818247334595328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019196.609171:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334595328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019196.609215:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019196.609219:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019196.609224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334595328:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019196.609231:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334595456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019196.609251:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019196.609259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019196/real 1734019196] req@ffff8800a934e300 x1818247334595456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019196.609270:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334595456/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019196.609278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019196.609280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019196.609283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334595456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019196.609288:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334595584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019196.609304:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019196.609310:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019196/real 1734019196] req@ffff8800a934dc00 x1818247334595584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019196.609321:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334595584/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019196.609328:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019196.609330:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019196.609333:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334595584:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019196.734738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019197.736743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019198.738728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019199.740734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019200.742752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019201.616616:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019201.616629:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019201.616635:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019201.616649:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3468 00000100:00080000:0.0:1734019201.616661:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019201.616671:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019201.616720:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334595712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019201.616759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019201.616764:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019201.616766:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019201.616771:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3468 00000100:00080000:0.0:1734019201.616774:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019201.616778:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019201.616789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019201.616801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334595840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019201.616811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019201.616814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019201.616816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019201.616821:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3468 00000100:00080000:0.0:1734019201.616824:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019201.616827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019201.616833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019201.616843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334595968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019201.616853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019201.616856:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019201.616858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019201.616862:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3468 00000100:00080000:0.0:1734019201.616864:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019201.616868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019201.616872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019201.616880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334596096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019201.616889:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019201.616892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019201.616894:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019201.616896:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334595712:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734019201.616898:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3468 00000100:00080000:0.0:1734019201.616900:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019201.616904:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019201.616908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019201.616917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334596224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019201.616925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019201.616928:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019201.616930:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019201.616934:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3468 00000100:00080000:0.0:1734019201.616937:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019201.616941:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019201.616945:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019201.616955:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334596352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019201.616963:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019201.616966:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019201.616968:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1734019201.616968:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734019201.616971:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3468 00000100:00080000:0.0:1734019201.616977:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019201.616980:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019201.616984:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019201.616993:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334596480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019201.616993:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019201/real 1734019201] req@ffff8800a934c380 x1818247334595712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019201.617014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334595712/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019201.617026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019201.617030:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019201.617035:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334595712:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019201.617044:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334595840:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019201.617065:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019201.617072:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019201/real 1734019201] req@ffff8800a934c700 x1818247334595840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019201.617084:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334595840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019201.617140:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019201.617143:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019201.617147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334595840:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019201.617155:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334595968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019201.617174:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019201.617181:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019201/real 1734019201] req@ffff8800a934ed80 x1818247334595968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019201.617192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334595968/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019201.617201:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019201.617203:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019201.617206:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334595968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019201.617233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334596096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019201.617246:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019201.617252:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019201/real 1734019201] req@ffff8800a934ca80 x1818247334596096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019201.617269:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334596096/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019201.617277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019201.617280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019201.617283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334596096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019201.617289:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334596224:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019201.617301:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019201.617308:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019201/real 1734019201] req@ffff8800a934ea00 x1818247334596224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019201.617318:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334596224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019201.617325:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019201.617327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019201.617330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334596224:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019201.617337:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334596352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019201.617349:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019201.617355:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019201/real 1734019201] req@ffff8800a934c000 x1818247334596352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019201.617365:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334596352/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019201.617372:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019201.617376:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019201.617379:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334596352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019201.617385:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334596480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019201.617398:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019201.617404:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019201/real 1734019201] req@ffff8800a934d180 x1818247334596480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019201.617414:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334596480/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019201.617421:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019201.617423:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019201.617426:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334596480:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019201.744723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019202.746731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019203.748739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019204.750757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019205.752776:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019206.624582:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019206.624592:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019206.624596:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019206.624606:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3473 00000100:00080000:0.0:1734019206.624616:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019206.624623:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019206.624658:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d880 x1818247334596608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019206.624698:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019206.624703:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019206.624705:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019206.624709:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3473 00000100:00080000:0.0:1734019206.624712:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019206.624716:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019206.624725:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019206.624736:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934df80 x1818247334596736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019206.624745:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019206.624748:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019206.624750:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019206.624754:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3473 00000100:00080000:0.0:1734019206.624757:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019206.624760:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019206.624765:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019206.624774:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334596864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019206.624783:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019206.624786:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019206.624787:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019206.624791:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3473 00000100:00080000:0.0:1734019206.624794:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019206.624797:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019206.624801:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019206.624804:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334596608:192.168.201.116@tcp:250:kworker.0 00000100:00100000:0.0:1734019206.624810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334596992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019206.624819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019206.624822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019206.624823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019206.624826:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3473 00000100:00080000:0.0:1734019206.624829:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019206.624832:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019206.624836:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019206.624843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334597120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019206.624851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019206.624854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019206.624855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019206.624858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3473 00000100:00080000:0.0:1734019206.624861:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019206.624864:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019206.624868:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019206.624871:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019206.624876:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334597248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019206.624884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019206.624888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019206.624890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019206.624893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3473 00000100:00100000:2.0:1734019206.624894:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019206/real 1734019206] req@ffff8800a934d880 x1818247334596608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019206.624896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019206.624899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019206.624903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019206.624910:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334597376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019206.624913:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d880 x1818247334596608/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019206.624923:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019206.624927:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019206.624932:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334596608:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019206.624939:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334596736:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019206.624955:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019206.624962:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019206/real 1734019206] req@ffff8800a934df80 x1818247334596736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019206.624973:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934df80 x1818247334596736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019206.624981:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019206.624984:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019206.624988:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334596736:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019206.624994:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334596864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019206.625049:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019206.625057:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019206/real 1734019206] req@ffff8800a934d180 x1818247334596864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019206.625068:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334596864/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019206.625076:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019206.625078:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019206.625081:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334596864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019206.625106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334596992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019206.625125:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019206.625132:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019206/real 1734019206] req@ffff8800a934c000 x1818247334596992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019206.625143:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334596992/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019206.625150:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019206.625152:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019206.625155:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334596992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019206.625161:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334597120:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019206.625173:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019206.625179:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019206/real 1734019206] req@ffff8800a934ea00 x1818247334597120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019206.625189:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334597120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019206.625196:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019206.625198:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019206.625201:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334597120:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019206.625206:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334597248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019206.625218:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019206.625224:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019206/real 1734019206] req@ffff8800a934ca80 x1818247334597248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019206.625240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334597248/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019206.625248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019206.625250:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019206.625253:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334597248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019206.625258:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334597376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019206.625270:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019206.625276:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019206/real 1734019206] req@ffff8800a934ed80 x1818247334597376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019206.625286:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334597376/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019206.625293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019206.625295:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019206.625298:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334597376:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019206.754733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019207.756730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019208.758732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019209.760735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019210.762746:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019211.632599:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019211.632612:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019211.632617:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019211.632631:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3478 00000100:00080000:0.0:1734019211.632644:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019211.632654:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019211.632703:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334597504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019211.632743:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019211.632747:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019211.632749:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019211.632755:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3478 00000100:00080000:0.0:1734019211.632758:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019211.632761:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019211.632773:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019211.632786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334597632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019211.632797:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019211.632801:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019211.632803:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019211.632807:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3478 00000100:00080000:0.0:1734019211.632809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019211.632813:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019211.632818:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019211.632827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334597760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019211.632836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019211.632839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019211.632841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019211.632845:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3478 00000100:00080000:0.0:1734019211.632847:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019211.632850:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019211.632855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019211.632863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334597888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019211.632873:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019211.632875:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334597504:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734019211.632876:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019211.632878:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019211.632882:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3478 00000100:00080000:0.0:1734019211.632885:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019211.632889:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019211.632893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019211.632902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334598016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019211.632910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019211.632913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019211.632914:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019211.632918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3478 00000100:00080000:0.0:1734019211.632959:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019211.632964:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019211.632969:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019211.632979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019211.632981:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334598144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019211.632989:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019211.632992:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019211.632994:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019211.632998:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3478 00000100:00080000:0.0:1734019211.633001:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019211.633005:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734019211.633005:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019211/real 1734019211] req@ffff8800a934c700 x1818247334597504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019211.633009:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019211.633018:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334598272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019211.633025:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334597504/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019211.633038:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019211.633042:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019211.633047:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334597504:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019211.633056:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334597632:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019211.633077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019211.633084:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019211/real 1734019211] req@ffff8800a934c380 x1818247334597632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019211.633095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334597632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019211.633104:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019211.633107:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019211.633110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334597632:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019211.633116:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334597760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019211.633130:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019211.633136:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019211/real 1734019211] req@ffff8800a934dc00 x1818247334597760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019211.633146:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334597760/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019211.633153:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019211.633156:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019211.633160:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334597760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019211.633186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334597888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019211.633201:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019211.633207:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019211/real 1734019211] req@ffff8800a934e300 x1818247334597888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019211.633217:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334597888/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019211.633225:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019211.633227:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019211.633230:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334597888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019211.633236:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334598016:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019211.633249:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019211.633255:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019211/real 1734019211] req@ffff8800a934f100 x1818247334598016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019211.633265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334598016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019211.633272:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019211.633275:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019211.633278:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334598016:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019211.633284:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334598144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019211.633300:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019211.633306:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019211/real 1734019211] req@ffff8800a934fb80 x1818247334598144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019211.633316:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334598144/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019211.633323:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019211.633325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019211.633328:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334598144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019211.633334:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334598272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019211.633347:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019211.633353:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019211/real 1734019211] req@ffff8800a934d500 x1818247334598272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019211.633363:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334598272/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019211.633370:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019211.633372:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019211.633375:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334598272:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019211.764731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019212.766736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019213.768750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019214.770728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019215.772761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019216.640622:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019216.640636:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019216.640641:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019216.640655:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3483 00000100:00080000:0.0:1734019216.640667:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019216.640677:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019216.640727:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e680 x1818247334598400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019216.640766:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019216.640770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019216.640772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019216.640777:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3483 00000100:00080000:0.0:1734019216.640780:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019216.640785:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019216.640797:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019216.640809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ce00 x1818247334598528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019216.640819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019216.640822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019216.640824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019216.640828:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3483 00000100:00080000:0.0:1734019216.640831:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019216.640868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019216.640876:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019216.640889:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d500 x1818247334598656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019216.640900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019216.640904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019216.640905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019216.640909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3483 00000100:00080000:0.0:1734019216.640912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019216.640915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019216.640920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019216.640929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934fb80 x1818247334598784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019216.640935:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334598400:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734019216.640940:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019216.640943:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019216.640945:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019216.640949:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3483 00000100:00080000:0.0:1734019216.640951:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019216.640954:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019216.640959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019216.640967:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934f100 x1818247334598912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019216.640975:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019216.640978:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019216.640980:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019216.640984:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3483 00000100:00080000:0.0:1734019216.640988:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019216.640991:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019216.640996:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019216.641005:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934e300 x1818247334599040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019216.641006:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:0.0:1734019216.641012:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019216.641016:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019216.641017:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019216.641021:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3483 00000100:00080000:0.0:1734019216.641024:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019216.641028:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:2.0:1734019216.641029:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019216/real 1734019216] req@ffff8800a934e680 x1818247334598400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019216.641032:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019216.641040:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934dc00 x1818247334599168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019216.641049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e680 x1818247334598400/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019216.641061:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019216.641065:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019216.641070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334598400:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019216.641078:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334598528:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019216.641096:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019216.641103:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019216/real 1734019216] req@ffff8800a934ce00 x1818247334598528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019216.641115:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ce00 x1818247334598528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019216.641123:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019216.641126:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019216.641129:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334598528:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019216.641135:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334598656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019216.641149:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019216.641155:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019216/real 1734019216] req@ffff8800a934d500 x1818247334598656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019216.641166:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d500 x1818247334598656/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019216.641174:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019216.641176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019216.641179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334598656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019216.641206:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334598784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019216.641219:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019216.641226:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019216/real 1734019216] req@ffff8800a934fb80 x1818247334598784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019216.641236:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934fb80 x1818247334598784/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019216.641243:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019216.641245:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019216.641248:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334598784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019216.641254:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334598912:192.168.201.116@tcp:38:kworker.0 00000800:00000100:2.0:1734019216.641267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019216.641273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019216/real 1734019216] req@ffff8800a934f100 x1818247334598912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019216.641283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934f100 x1818247334598912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019216.641290:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019216.641292:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019216.641295:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334598912:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019216.641302:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334599040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019216.641315:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019216.641321:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019216/real 1734019216] req@ffff8800a934e300 x1818247334599040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019216.641331:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934e300 x1818247334599040/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019216.641338:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019216.641341:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019216.641344:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334599040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019216.641349:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334599168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:2.0:1734019216.641362:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019216.641368:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019216/real 1734019216] req@ffff8800a934dc00 x1818247334599168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019216.641377:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934dc00 x1818247334599168/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019216.641384:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019216.641387:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:2.0:1734019216.641390:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334599168:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019216.774749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019217.776760:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019218.778734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019219.780737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019220.782729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019221.648698:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019221.648708:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019221.648713:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019221.648723:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3488 00000100:00080000:2.0:1734019221.648731:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019221.648739:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019221.648810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334599296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019221.648848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019221.648852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019221.648854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019221.648858:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3488 00000100:00080000:2.0:1734019221.648862:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019221.648866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019221.648877:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019221.648889:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334599424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019221.648893:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334599296:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019221.648900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019221.648903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019221.648905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019221.648908:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3488 00000100:00080000:2.0:1734019221.648911:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019221.648914:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019221.648920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019221.648928:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334599552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019221.648936:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019221.648939:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019221.648941:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019221.648944:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3488 00000100:00080000:2.0:1734019221.648947:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019221.648950:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019221.648954:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000800:00000100:0.0:1734019221.648957:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019221.648962:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334599680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019221.648970:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019221.648972:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019221.648974:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019221.648977:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3488 00000100:00080000:2.0:1734019221.648980:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019221.648981:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019221/real 1734019221] req@ffff8800a78a0700 x1818247334599296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019221.648983:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019221.648987:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019221.648994:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334599808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019221.649001:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334599296/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019221.649003:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019221.649006:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019221.649007:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019221.649011:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3488 00000100:00080000:0.0:1734019221.649012:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019221.649014:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019221.649016:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019221.649017:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734019221.649021:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334599296:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019221.649021:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019221.649031:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334599936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019221.649044:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334599424:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019221.649044:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019221.649049:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019221.649051:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019221.649057:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3488 00000100:00080000:2.0:1734019221.649061:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734019221.649062:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019221.649066:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734019221.649070:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019221/real 1734019221] req@ffff8800a78a3480 x1818247334599424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019221.649073:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019221.649082:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334599424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019221.649087:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334600064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019221.649090:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019221.649092:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019221.649096:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334599424:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019221.649102:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334599552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019221.649115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019221.649122:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019221/real 1734019221] req@ffff8800a78a1f80 x1818247334599552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019221.649132:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334599552/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019221.649139:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019221.649142:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019221.649145:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334599552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019221.649151:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334599680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019221.649167:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019221.649173:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019221/real 1734019221] req@ffff8800a78a0a80 x1818247334599680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019221.649184:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334599680/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019221.649191:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019221.649193:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019221.649196:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334599680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019221.649201:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334599808:192.168.201.116@tcp:38:kworker.0 00000800:00000100:0.0:1734019221.649214:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019221.649220:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019221/real 1734019221] req@ffff8800a78a2680 x1818247334599808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019221.649230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334599808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019221.649237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019221.649239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019221.649242:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334599808:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019221.649258:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334599936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019221.649270:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019221.649277:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019221/real 1734019221] req@ffff8800a78a1880 x1818247334599936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019221.649294:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334599936/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019221.649301:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019221.649303:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019221.649306:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334599936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019221.649311:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334600064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:0.0:1734019221.649323:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:0.0:1734019221.649329:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019221/real 1734019221] req@ffff8800a78a3b80 x1818247334600064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019221.649339:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334600064/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019221.649346:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019221.649348:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:0.0:1734019221.649351:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334600064:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019221.784727:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019222.786769:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019223.788744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019224.790757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019225.792752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019226.656621:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019226.656630:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019226.656635:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019226.656645:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3493 00000100:00080000:2.0:1734019226.656655:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019226.656662:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019226.656737:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334600192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019226.656770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019226.656774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019226.656776:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019226.656781:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3493 00000100:00080000:2.0:1734019226.656785:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019226.656788:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019226.656798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019226.656810:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334600320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019226.656820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019226.656823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019226.656825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019226.656828:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3493 00000100:00080000:2.0:1734019226.656831:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019226.656834:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019226.656839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019226.656848:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334600448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019226.656856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019226.656862:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019226.656864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019226.656867:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3493 00000100:00080000:2.0:1734019226.656870:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019226.656873:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:0.0:1734019226.656874:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334600192:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019226.656878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019226.656886:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334600576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019226.656895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019226.656898:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019226.656899:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019226.656902:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3493 00000100:00080000:2.0:1734019226.656905:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019226.656908:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019226.656912:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019226.656919:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334600704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019226.656927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019226.656930:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019226.656931:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019226.656934:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3493 00000100:00080000:2.0:1734019226.656937:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019226.656941:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019226.656945:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019226.656953:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334600832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019226.656961:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019226.656965:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734019226.656966:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334600320:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019226.656966:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019226.656970:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3493 00000100:00080000:2.0:1734019226.656972:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019226.656975:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019226.656979:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019226.656986:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334600960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019226.656987:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334600448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019226.657010:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334600576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019226.657024:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334600704:192.168.201.116@tcp:38:kworker.0 00000100:00100000:0.0:1734019226.657039:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334600832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019226.657053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334600960:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019226.794758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019227.796738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019228.798774:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019229.662783:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019229.662830:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019229.662848:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019229.662888:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019229.662895:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019229.662902:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019229.662909:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019229.662915:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019229.662921:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734019229.662965:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019226/real 1734019229] req@ffff8800a78a0a80 x1818247334600576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019229.663006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334600576/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019229.663019:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019229.663024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019229.663029:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334600576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019229.663038:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019226/real 1734019229] req@ffff8800a78a1f80 x1818247334600704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019229.663051:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334600704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019229.663059:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019229.663062:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019229.663065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334600704:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019229.663072:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019226/real 1734019229] req@ffff8800a78a3480 x1818247334600832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019229.663082:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334600832/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019229.663090:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019229.663093:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019229.663096:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334600832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019229.663102:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019226/real 1734019229] req@ffff8800a78a0700 x1818247334600960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019229.663112:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334600960/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019229.663122:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019229.663126:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019229.663132:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334600960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019229.663141:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019226/real 1734019229] req@ffff8800a78a3b80 x1818247334600192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019229.663159:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334600192/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019229.663170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019229.663174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019229.663178:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334600192:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019229.663188:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019226/real 1734019229] req@ffff8800a78a1880 x1818247334600320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019229.663205:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334600320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019229.663216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019229.663220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019229.663225:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334600320:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019229.663235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019226/real 1734019229] req@ffff8800a78a2680 x1818247334600448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019229.663252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334600448/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019229.663264:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019229.663268:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019229.663273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334600448:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019229.800773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019230.802849:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019231.664645:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019231.664654:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019231.664659:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019231.664669:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3498 00000100:00080000:2.0:1734019231.664678:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019231.664685:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019231.664722:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334601088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019231.664762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019231.664766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019231.664768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019231.664772:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3498 00000100:00080000:2.0:1734019231.664776:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019231.664780:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019231.664789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019231.664800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334601216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019231.664809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019231.664812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019231.664814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019231.664817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3498 00000100:00080000:2.0:1734019231.664820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019231.664824:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019231.664828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019231.664837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334601344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019231.664845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019231.664848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019231.664850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019231.664853:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3498 00000100:00080000:2.0:1734019231.664856:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019231.664859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019231.664863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019231.664871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334601472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019231.664879:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334601088:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019231.664880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019231.664883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019231.664884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019231.664888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3498 00000100:00080000:2.0:1734019231.664890:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019231.664893:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019231.664897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019231.664904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334601600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019231.664912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019231.664915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019231.664916:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019231.664920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3498 00000100:00080000:2.0:1734019231.664922:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019231.664925:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019231.664929:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019231.664936:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334601728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019231.664944:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734019231.664946:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019231.664948:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019231.664949:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019231.664953:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3498 00000100:00080000:2.0:1734019231.664956:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019231.664959:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019231.664962:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019231.664967:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019231/real 1734019231] req@ffff8800a78a2680 x1818247334601088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019231.664970:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334601856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019231.664986:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334601088/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019231.664997:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019231.665000:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019231.665005:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334601088:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019231.665013:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334601216:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019231.665029:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019231.665036:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019231/real 1734019231] req@ffff8800a78a1880 x1818247334601216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019231.665047:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334601216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019231.665055:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019231.665058:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019231.665061:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334601216:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019231.665067:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334601344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019231.665081:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019231.665087:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019231/real 1734019231] req@ffff8800a78a3b80 x1818247334601344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019231.665105:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334601344/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019231.665113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019231.665115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019231.665118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334601344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019231.665142:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334601472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019231.665155:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019231.665161:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019231/real 1734019231] req@ffff8800a78a0700 x1818247334601472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019231.665171:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334601472/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019231.665178:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019231.665180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019231.665184:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334601472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019231.665189:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334601600:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019231.665201:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019231.665207:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019231/real 1734019231] req@ffff8800a78a3480 x1818247334601600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019231.665217:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334601600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019231.665225:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019231.665228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019231.665231:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334601600:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019231.665237:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334601728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019231.665249:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019231.665255:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019231/real 1734019231] req@ffff8800a78a1f80 x1818247334601728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019231.665264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334601728/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019231.665272:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019231.665274:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019231.665277:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334601728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019231.665282:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334601856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019231.665294:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019231.665300:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019231/real 1734019231] req@ffff8800a78a0a80 x1818247334601856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019231.665310:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334601856/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019231.665316:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019231.665319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019231.665322:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334601856:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019231.804702:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019232.806657:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019233.808679:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019234.810622:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019235.812674:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019236.672618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019236.672628:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019236.672632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019236.672642:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3503 00000100:00080000:2.0:1734019236.672650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019236.672657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019236.672695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334601984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019236.672727:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019236.672731:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019236.672733:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019236.672737:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3503 00000100:00080000:2.0:1734019236.672741:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019236.672744:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019236.672754:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019236.672765:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334602112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019236.672776:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019236.672779:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019236.672781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019236.672785:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3503 00000100:00080000:2.0:1734019236.672787:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019236.672791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019236.672795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019236.672803:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334602240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019236.672812:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019236.672815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019236.672817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019236.672820:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3503 00000100:00080000:2.0:1734019236.672822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019236.672826:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019236.672830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019236.672832:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334601984:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019236.672838:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334602368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019236.672847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019236.672850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019236.672853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019236.672856:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3503 00000100:00080000:2.0:1734019236.672859:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019236.672862:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019236.672866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019236.672873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334602496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019236.672881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019236.672884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019236.672886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019236.672889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3503 00000100:00080000:2.0:1734019236.672891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019236.672895:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019236.672899:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019236.672899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019236.672906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334602624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019236.672914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019236.672917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019236.672919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019236.672921:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019236/real 1734019236] req@ffff8800a78a0a80 x1818247334601984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019236.672922:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3503 00000100:00080000:2.0:1734019236.672925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019236.672928:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019236.672932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019236.672939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334602752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019236.672941:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334601984/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019236.672951:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019236.672955:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019236.672959:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334601984:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019236.672967:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334602112:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019236.672984:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019236.672991:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019236/real 1734019236] req@ffff8800a78a1f80 x1818247334602112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019236.673002:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334602112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019236.673010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019236.673013:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019236.673016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334602112:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019236.673022:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334602240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019236.673035:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019236.673041:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019236/real 1734019236] req@ffff8800a78a3480 x1818247334602240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019236.673059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334602240/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019236.673067:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019236.673069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019236.673072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334602240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019236.673096:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334602368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019236.673109:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019236.673116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019236/real 1734019236] req@ffff8800a78a0700 x1818247334602368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019236.673126:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334602368/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019236.673133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019236.673135:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019236.673138:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334602368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019236.673144:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334602496:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019236.673156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019236.673162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019236/real 1734019236] req@ffff8800a78a3b80 x1818247334602496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019236.673172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334602496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019236.673179:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019236.673182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019236.673185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334602496:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019236.673191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334602624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019236.673203:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019236.673209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019236/real 1734019236] req@ffff8800a78a1880 x1818247334602624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019236.673219:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334602624/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019236.673226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019236.673228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019236.673231:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334602624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019236.673237:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334602752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019236.673249:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019236.673255:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019236/real 1734019236] req@ffff8800a78a2680 x1818247334602752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019236.673264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334602752/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019236.673271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019236.673274:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019236.673277:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334602752:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019236.814751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019237.816718:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019238.818733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019239.820813:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019240.822744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019241.680631:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019241.680643:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019241.680648:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019241.680662:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3508 00000100:00080000:2.0:1734019241.680673:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019241.680683:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019241.680734:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334602880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019241.680773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019241.680778:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019241.680780:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019241.680785:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3508 00000100:00080000:2.0:1734019241.680789:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019241.680793:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019241.680804:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019241.680817:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334603008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019241.680827:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019241.680830:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019241.680832:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019241.680836:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3508 00000100:00080000:2.0:1734019241.680840:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019241.680844:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019241.680849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019241.680859:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334603136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019241.680868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019241.680871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019241.680873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019241.680876:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3508 00000100:00080000:2.0:1734019241.680879:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019241.680882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019241.680887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019241.680896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334603264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019241.680905:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019241.680909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019241.680910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019241.680914:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3508 00000100:00080000:2.0:1734019241.680916:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019241.680920:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019241.680921:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334602880:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019241.680924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019241.680933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334603392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019241.680947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019241.680950:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019241.680952:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019241.680956:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3508 00000100:00080000:2.0:1734019241.680958:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019241.680962:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019241.680966:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019241.680974:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334603520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019241.680982:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019241.680986:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019241.680987:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019241.680990:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3508 00000100:00080000:2.0:1734019241.680994:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019241.680997:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019241.680997:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019241.681001:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019241.681009:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334603648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019241.681020:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019241/real 1734019241] req@ffff8800a78a2680 x1818247334602880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019241.681040:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334602880/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019241.681052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019241.681056:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019241.681061:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334602880:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019241.681070:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334603008:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019241.681088:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019241.681096:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019241/real 1734019241] req@ffff8800a78a1880 x1818247334603008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019241.681107:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334603008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019241.681115:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019241.681117:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019241.681121:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334603008:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019241.681127:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334603136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019241.681141:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019241.681147:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019241/real 1734019241] req@ffff8800a78a3b80 x1818247334603136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019241.681165:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334603136/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019241.681173:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019241.681176:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019241.681179:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334603136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019241.681205:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334603264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019241.681219:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019241.681225:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019241/real 1734019241] req@ffff8800a78a0700 x1818247334603264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019241.681235:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334603264/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019241.681243:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019241.681245:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019241.681249:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334603264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019241.681255:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334603392:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019241.681268:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019241.681274:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019241/real 1734019241] req@ffff8800a78a3480 x1818247334603392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019241.681284:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334603392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019241.681291:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019241.681294:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019241.681297:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334603392:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019241.681303:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334603520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019241.681315:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019241.681321:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019241/real 1734019241] req@ffff8800a78a1f80 x1818247334603520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019241.681331:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334603520/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019241.681338:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019241.681340:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019241.681343:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334603520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019241.681349:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334603648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019241.681362:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019241.681368:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019241/real 1734019241] req@ffff8800a78a0a80 x1818247334603648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019241.681377:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334603648/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019241.681384:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019241.681387:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019241.681389:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334603648:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019241.824748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019242.826712:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019243.828606:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019244.830707:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019245.832680:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019246.688566:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019246.688574:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019246.688577:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019246.688585:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3513 00000100:00080000:2.0:1734019246.688592:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019246.688597:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019246.688625:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334603776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019246.688646:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019246.688648:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019246.688649:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019246.688651:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3513 00000100:00080000:2.0:1734019246.688653:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019246.688655:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019246.688661:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019246.688667:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334603904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019246.688672:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019246.688674:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019246.688674:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019246.688677:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3513 00000100:00080000:2.0:1734019246.688678:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019246.688680:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019246.688682:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019246.688687:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334604032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019246.688691:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019246.688693:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019246.688694:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019246.688696:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3513 00000100:00080000:2.0:1734019246.688697:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019246.688699:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019246.688701:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019246.688705:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334604160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019246.688710:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019246.688711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019246.688712:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019246.688714:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3513 00000100:00080000:2.0:1734019246.688715:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019246.688717:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019246.688719:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334603776:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019246.688719:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019246.688723:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334604288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019246.688727:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019246.688729:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019246.688729:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019246.688731:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3513 00000100:00080000:2.0:1734019246.688733:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019246.688734:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019246.688737:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019246.688741:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334604416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019246.688746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019246.688747:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019246.688748:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019246.688750:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3513 00000100:00080000:2.0:1734019246.688752:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019246.688754:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019246.688756:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019246.688759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334604544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019246.688763:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019246.688776:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019246/real 1734019246] req@ffff8800a78a0a80 x1818247334603776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019246.688788:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334603776/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019246.688794:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019246.688796:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019246.688798:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334603776:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019246.688802:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334603904:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019246.688811:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019246.688815:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019246/real 1734019246] req@ffff8800a78a1f80 x1818247334603904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019246.688820:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334603904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019246.688824:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019246.688826:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019246.688827:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334603904:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019246.688830:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334604032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019246.688838:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019246.688841:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019246/real 1734019246] req@ffff8800a78a3480 x1818247334604032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019246.688847:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334604032/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019246.688851:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019246.688852:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019246.688854:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334604032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019246.688857:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334604160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019246.688863:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019246.688866:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019246/real 1734019246] req@ffff8800a78a0700 x1818247334604160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019246.688871:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334604160/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019246.688875:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019246.688876:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019246.688877:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334604160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019246.688892:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334604288:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019246.688898:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019246.688901:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019246/real 1734019246] req@ffff8800a78a3b80 x1818247334604288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019246.688907:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334604288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019246.688911:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019246.688912:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019246.688914:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334604288:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019246.688917:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334604416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019246.688923:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019246.688926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019246/real 1734019246] req@ffff8800a78a1880 x1818247334604416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019246.688931:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334604416/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019246.688935:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019246.688936:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019246.688937:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334604416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019246.688940:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334604544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019246.688946:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019246.688949:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019246/real 1734019246] req@ffff8800a78a2680 x1818247334604544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019246.688954:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334604544/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019246.688958:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019246.688959:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019246.688960:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334604544:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019246.834718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019247.836751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019248.838665:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019249.840640:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019250.842721:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019251.696612:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019251.696624:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019251.696630:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019251.696644:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3518 00000100:00080000:2.0:1734019251.696655:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019251.696665:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019251.696716:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334604672/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019251.696755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019251.696763:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019251.696765:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019251.696770:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3518 00000100:00080000:2.0:1734019251.696773:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019251.696777:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019251.696788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019251.696801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334604800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019251.696811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019251.696814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019251.696816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019251.696820:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3518 00000100:00080000:2.0:1734019251.696822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019251.696827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019251.696832:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019251.696841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334604928/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019251.696850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019251.696853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019251.696855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019251.696858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3518 00000100:00080000:2.0:1734019251.696861:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019251.696864:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019251.696869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019251.696878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334605056/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019251.696888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019251.696891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019251.696893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019251.696897:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3518 00000100:00100000:1.0:1734019251.696898:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334604672:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019251.696899:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019251.696902:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019251.696907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019251.696915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334605184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019251.696923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019251.696926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019251.696928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019251.696931:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3518 00000100:00080000:2.0:1734019251.696934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019251.696937:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019251.696941:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019251.696949:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334605312/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019251.696957:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019251.696961:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019251.696963:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019251.696967:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3518 00000100:00080000:2.0:1734019251.696971:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019251.696972:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019251.696975:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019251.696979:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019251.696986:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334605440/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019251.696996:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019251/real 1734019251] req@ffff8800a78a2680 x1818247334604672/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019251.697016:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334604672/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019251.697029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019251.697032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019251.697038:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334604672:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019251.697046:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334604800:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019251.697064:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019251.697071:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019251/real 1734019251] req@ffff8800a78a1880 x1818247334604800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019251.697082:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334604800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019251.697090:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019251.697092:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019251.697096:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334604800:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019251.697102:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334604928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019251.697116:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019251.697122:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019251/real 1734019251] req@ffff8800a78a3b80 x1818247334604928/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019251.697140:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334604928/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019251.697148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019251.697151:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019251.697154:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334604928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019251.697179:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334605056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019251.697193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019251.697200:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019251/real 1734019251] req@ffff8800a78a0700 x1818247334605056/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019251.697210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334605056/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019251.697218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019251.697220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019251.697223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334605056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019251.697230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334605184:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019251.697282:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019251.697290:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019251/real 1734019251] req@ffff8800a78a3480 x1818247334605184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019251.697301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334605184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019251.697309:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019251.697312:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019251.697315:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334605184:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019251.697322:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334605312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019251.697336:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019251.697342:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019251/real 1734019251] req@ffff8800a78a1f80 x1818247334605312/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019251.697352:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334605312/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019251.697360:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019251.697363:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019251.697366:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334605312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019251.697371:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334605440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019251.697384:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019251.697390:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019251/real 1734019251] req@ffff8800a78a0a80 x1818247334605440/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019251.697400:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334605440/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019251.697407:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019251.697410:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019251.697413:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334605440:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019251.844723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019252.846738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019253.848768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019254.850729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019255.852756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019256.704609:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019256.704620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019256.704625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019256.704635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3523 00000100:00080000:2.0:1734019256.704644:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019256.704651:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019256.704690:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334605568/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019256.704724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019256.704729:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019256.704731:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019256.704735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3523 00000100:00080000:2.0:1734019256.704738:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019256.704742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019256.704751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019256.704763:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334605696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019256.704773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019256.704777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019256.704778:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019256.704782:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3523 00000100:00080000:2.0:1734019256.704785:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019256.704788:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019256.704793:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019256.704801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334605824/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019256.704810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019256.704813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019256.704814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019256.704818:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3523 00000100:00080000:2.0:1734019256.704820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019256.704823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019256.704827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019256.704836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334605952/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019256.704838:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334605568:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019256.704845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019256.704848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019256.704849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019256.704852:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3523 00000100:00080000:2.0:1734019256.704855:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019256.704858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019256.704862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019256.704870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334606080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019256.704878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019256.704880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019256.704882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019256.704885:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3523 00000100:00080000:2.0:1734019256.704888:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019256.704891:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019256.704895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019256.704903:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334606208/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019256.704905:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019256.704910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019256.704913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019256.704915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019256.704919:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3523 00000100:00080000:2.0:1734019256.704921:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019256.704925:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019256.704926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019256/real 1734019256] req@ffff8800a78a0a80 x1818247334605568/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019256.704928:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019256.704936:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334606336/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019256.704946:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334605568/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019256.704957:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019256.704960:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019256.704965:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334605568:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019256.704972:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334605696:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019256.704989:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019256.704997:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019256/real 1734019256] req@ffff8800a78a1f80 x1818247334605696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019256.705008:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334605696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019256.705016:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019256.705018:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019256.705022:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334605696:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019256.705027:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334605824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019256.705041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019256.705047:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019256/real 1734019256] req@ffff8800a78a3480 x1818247334605824/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019256.705058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334605824/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019256.705066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019256.705069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019256.705072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334605824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019256.705095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334605952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019256.705108:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019256.705115:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019256/real 1734019256] req@ffff8800a78a0700 x1818247334605952/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019256.705124:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334605952/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019256.705132:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019256.705134:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019256.705137:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334605952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019256.705142:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334606080:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019256.705198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019256.705207:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019256/real 1734019256] req@ffff8800a78a3b80 x1818247334606080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019256.705218:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334606080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019256.705225:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019256.705228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019256.705232:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334606080:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019256.705238:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334606208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019256.705253:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019256.705259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019256/real 1734019256] req@ffff8800a78a1880 x1818247334606208/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019256.705270:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334606208/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019256.705278:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019256.705280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019256.705283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334606208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019256.705288:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334606336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019256.705300:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019256.705306:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019256/real 1734019256] req@ffff8800a78a2680 x1818247334606336/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019256.705316:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334606336/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019256.705323:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019256.705325:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019256.705328:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334606336:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019256.854722:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019257.856696:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019258.858701:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019259.860772:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019260.862703:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019261.712623:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019261.712634:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019261.712638:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019261.712649:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3528 00000100:00080000:2.0:1734019261.712657:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019261.712665:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019261.712702:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334606464/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019261.712735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019261.712739:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019261.712741:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019261.712746:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3528 00000100:00080000:2.0:1734019261.712749:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019261.712752:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019261.712762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019261.712773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334606592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019261.712782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019261.712786:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019261.712787:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019261.712791:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3528 00000100:00080000:2.0:1734019261.712795:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019261.712798:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019261.712802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019261.712811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334606720/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019261.712819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019261.712823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019261.712824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019261.712827:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3528 00000100:00080000:2.0:1734019261.712830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019261.712833:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019261.712837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019261.712845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334606848/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019261.712850:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334606464:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019261.712854:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019261.712857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019261.712859:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019261.712862:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3528 00000100:00080000:2.0:1734019261.712865:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019261.712868:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019261.712872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019261.712879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334606976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019261.712893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019261.712896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019261.712898:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019261.712901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3528 00000100:00080000:2.0:1734019261.712904:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019261.712907:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019261.712911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019261.712914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019261.712918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334607104/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019261.712926:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019261.712929:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019261.712930:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019261.712933:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3528 00000100:00100000:1.0:1734019261.712936:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019261/real 1734019261] req@ffff8800a78a2680 x1818247334606464/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019261.712936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019261.712939:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019261.712943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019261.712950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334607232/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019261.712955:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334606464/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019261.712966:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019261.712969:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019261.712974:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334606464:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019261.712981:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334606592:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019261.712998:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019261.713005:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019261/real 1734019261] req@ffff8800a78a1880 x1818247334606592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019261.713016:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334606592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019261.713024:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019261.713026:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019261.713030:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334606592:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019261.713036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334606720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019261.713049:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019261.713055:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019261/real 1734019261] req@ffff8800a78a3b80 x1818247334606720/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019261.713118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334606720/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019261.713126:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019261.713128:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019261.713132:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334606720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019261.713157:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334606848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019261.713176:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019261.713183:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019261/real 1734019261] req@ffff8800a78a0700 x1818247334606848/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019261.713194:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334606848/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019261.713202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019261.713204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019261.713207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334606848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019261.713213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334606976:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019261.713225:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019261.713231:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019261/real 1734019261] req@ffff8800a78a3480 x1818247334606976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019261.713241:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334606976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019261.713249:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019261.713252:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019261.713255:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334606976:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019261.713261:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334607104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019261.713272:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019261.713278:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019261/real 1734019261] req@ffff8800a78a1f80 x1818247334607104/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019261.713288:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334607104/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019261.713295:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019261.713297:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019261.713300:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334607104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019261.713305:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334607232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019261.713317:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019261.713323:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019261/real 1734019261] req@ffff8800a78a0a80 x1818247334607232/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019261.713332:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334607232/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019261.713339:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019261.713341:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019261.713344:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334607232:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019261.864706:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019262.866704:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019263.868762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019264.870712:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019265.872799:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019266.720618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019266.720630:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019266.720636:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019266.720650:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3533 00000100:00080000:2.0:1734019266.720661:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019266.720670:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019266.720721:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334607360/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019266.720760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019266.720765:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019266.720767:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019266.720772:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3533 00000100:00080000:2.0:1734019266.720775:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019266.720779:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019266.720791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019266.720803:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334607488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019266.720813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019266.720816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019266.720818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019266.720822:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3533 00000100:00080000:2.0:1734019266.720825:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019266.720829:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019266.720834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019266.720843:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334607616/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019266.720852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019266.720857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019266.720859:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019266.720862:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3533 00000100:00080000:2.0:1734019266.720865:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019266.720868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019266.720873:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019266.720881:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334607744/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019266.720892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019266.720895:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019266.720897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019266.720900:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3533 00000100:00080000:2.0:1734019266.720903:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019266.720907:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019266.720911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019266.720916:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334607360:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019266.720919:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334607872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019266.720927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019266.720930:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019266.720932:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019266.720935:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3533 00000100:00080000:2.0:1734019266.720938:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019266.720942:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019266.720947:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019266.720955:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334608000/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019266.720963:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019266.720966:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019266.720968:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019266.720971:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3533 00000100:00080000:2.0:1734019266.720974:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019266.720978:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019266.720982:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019266.721026:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334608128/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019266.721037:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019266.721064:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019266/real 1734019266] req@ffff8800a78a0a80 x1818247334607360/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019266.721084:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334607360/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019266.721095:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019266.721100:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019266.721104:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334607360:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019266.721113:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334607488:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019266.721133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019266.721140:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019266/real 1734019266] req@ffff8800a78a1f80 x1818247334607488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019266.721151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334607488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019266.721160:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019266.721163:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019266.721166:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334607488:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019266.721173:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334607616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019266.721187:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019266.721193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019266/real 1734019266] req@ffff8800a78a3480 x1818247334607616/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019266.721204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334607616/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019266.721212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019266.721215:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019266.721218:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334607616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019266.721244:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334607744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019266.721258:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019266.721264:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019266/real 1734019266] req@ffff8800a78a0700 x1818247334607744/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019266.721274:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334607744/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019266.721282:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019266.721285:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019266.721288:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334607744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019266.721294:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334607872:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019266.721307:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019266.721313:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019266/real 1734019266] req@ffff8800a78a3b80 x1818247334607872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019266.721323:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334607872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019266.721331:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019266.721334:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019266.721337:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334607872:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019266.721343:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334608000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019266.721356:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019266.721362:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019266/real 1734019266] req@ffff8800a78a1880 x1818247334608000/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019266.721372:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334608000/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019266.721380:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019266.721382:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019266.721386:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334608000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019266.721391:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334608128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019266.721404:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019266.721410:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019266/real 1734019266] req@ffff8800a78a2680 x1818247334608128/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019266.721420:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334608128/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019266.721427:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019266.721429:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019266.721433:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334608128:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019266.874707:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019267.876734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019268.878724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019269.880740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019270.882720:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019271.728604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019271.728614:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019271.728619:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019271.728629:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3538 00000100:00080000:2.0:1734019271.728638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019271.728645:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019271.728682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334608256/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019271.728717:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019271.728723:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019271.728726:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019271.728730:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3538 00000100:00080000:2.0:1734019271.728733:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019271.728737:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019271.728746:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019271.728757:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334608384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019271.728767:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019271.728770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019271.728772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019271.728775:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3538 00000100:00080000:2.0:1734019271.728778:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019271.728782:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019271.728786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019271.728795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334608512/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019271.728803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019271.728806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019271.728808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019271.728811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3538 00000100:00080000:2.0:1734019271.728813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019271.728817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019271.728821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019271.728830:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334608256:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019271.728830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334608640/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019271.728839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019271.728842:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019271.728844:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019271.728847:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3538 00000100:00080000:2.0:1734019271.728850:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019271.728853:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019271.728856:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019271.728864:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334608768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019271.728872:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019271.728875:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019271.728876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019271.728879:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3538 00000100:00080000:2.0:1734019271.728891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019271.728895:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019271.728897:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019271.728934:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019271.728945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334608896/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019271.728953:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019271/real 1734019271] req@ffff8800a78a2680 x1818247334608256/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019271.728954:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019271.728957:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019271.728959:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019271.728963:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3538 00000100:00080000:2.0:1734019271.728967:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019271.728970:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019271.728973:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334608256/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019271.728974:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019271.728981:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334609024/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019271.728983:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019271.728987:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019271.728992:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334608256:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019271.729000:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334608384:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019271.729020:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019271.729027:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019271/real 1734019271] req@ffff8800a78a1880 x1818247334608384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019271.729038:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334608384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019271.729046:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019271.729049:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019271.729052:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334608384:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019271.729058:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334608512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019271.729071:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019271.729077:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019271/real 1734019271] req@ffff8800a78a3b80 x1818247334608512/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019271.729096:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334608512/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019271.729103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019271.729105:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019271.729108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334608512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019271.729133:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334608640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019271.729147:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019271.729153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019271/real 1734019271] req@ffff8800a78a0700 x1818247334608640/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019271.729163:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334608640/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019271.729170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019271.729172:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019271.729175:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334608640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019271.729181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334608768:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019271.729193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019271.729199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019271/real 1734019271] req@ffff8800a78a3480 x1818247334608768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019271.729209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334608768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019271.729216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019271.729220:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019271.729223:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334608768:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019271.729229:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334608896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019271.729240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019271.729246:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019271/real 1734019271] req@ffff8800a78a1f80 x1818247334608896/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019271.729256:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334608896/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019271.729263:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019271.729265:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019271.729268:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334608896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019271.729274:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334609024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019271.729286:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019271.729291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019271/real 1734019271] req@ffff8800a78a0a80 x1818247334609024/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019271.729301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334609024/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019271.729308:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019271.729310:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019271.729313:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334609024:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019271.884670:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019272.886728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019273.888734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019274.890750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019275.892800:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019276.736580:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019276.736589:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019276.736594:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019276.736604:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3543 00000100:00080000:2.0:1734019276.736613:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019276.736621:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019276.736657:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334609152/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019276.736691:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019276.736696:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019276.736698:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019276.736702:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3543 00000100:00080000:2.0:1734019276.736705:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019276.736709:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019276.736719:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019276.736730:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334609280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019276.736740:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019276.736744:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019276.736745:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019276.736749:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3543 00000100:00080000:2.0:1734019276.736752:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019276.736755:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019276.736760:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019276.736768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334609408/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019276.736777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019276.736780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019276.736781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019276.736785:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3543 00000100:00080000:2.0:1734019276.736787:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019276.736790:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019276.736794:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019276.736801:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334609152:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019276.736803:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334609536/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019276.736812:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019276.736849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019276.736851:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019276.736854:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3543 00000100:00080000:2.0:1734019276.736858:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019276.736861:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019276.736866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019276.736875:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334609664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019276.736884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019276.736887:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019276.736889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019276.736892:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3543 00000100:00080000:2.0:1734019276.736895:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019276.736896:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019276.736899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019276.736902:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019276.736910:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334609792/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019276.736918:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019276/real 1734019276] req@ffff8800a78a0a80 x1818247334609152/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019276.736918:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019276.736921:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019276.736922:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019276.736926:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3543 00000100:00080000:2.0:1734019276.736928:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019276.736931:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019276.736935:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019276.736938:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334609152/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019276.736943:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334609920/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019276.736948:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019276.736952:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019276.736957:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334609152:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019276.736964:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334609280:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019276.736982:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019276.736989:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019276/real 1734019276] req@ffff8800a78a1f80 x1818247334609280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019276.737000:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334609280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019276.737008:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019276.737010:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019276.737014:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334609280:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019276.737020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334609408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019276.737032:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019276.737038:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019276/real 1734019276] req@ffff8800a78a3480 x1818247334609408/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019276.737050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334609408/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019276.737057:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019276.737060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019276.737063:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334609408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019276.737087:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334609536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019276.737100:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019276.737106:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019276/real 1734019276] req@ffff8800a78a0700 x1818247334609536/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019276.737116:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334609536/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019276.737124:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019276.737126:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019276.737129:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334609536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019276.737135:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334609664:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019276.737147:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019276.737153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019276/real 1734019276] req@ffff8800a78a3b80 x1818247334609664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019276.737163:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334609664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019276.737171:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019276.737173:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019276.737176:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334609664:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019276.737182:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334609792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019276.737193:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019276.737199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019276/real 1734019276] req@ffff8800a78a1880 x1818247334609792/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019276.737209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334609792/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019276.737216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019276.737218:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019276.737221:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334609792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019276.737227:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334609920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019276.737239:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019276.737245:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019276/real 1734019276] req@ffff8800a78a2680 x1818247334609920/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019276.737255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334609920/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019276.737262:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019276.737264:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019276.737267:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334609920:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019276.894715:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019277.896744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019278.898733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019279.900802:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019280.902731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019281.744597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019281.744606:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019281.744611:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019281.744621:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3548 00000100:00080000:2.0:1734019281.744630:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019281.744637:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019281.744676:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334610048/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019281.744711:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019281.744715:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019281.744717:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019281.744721:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3548 00000100:00080000:2.0:1734019281.744724:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019281.744767:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019281.744778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019281.744791:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334610176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019281.744802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019281.744805:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019281.744807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019281.744811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3548 00000100:00080000:2.0:1734019281.744814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019281.744818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019281.744823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019281.744832:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334610304/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019281.744834:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334610048:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019281.744842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019281.744845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019281.744846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019281.744850:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3548 00000100:00080000:2.0:1734019281.744852:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019281.744856:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019281.744860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019281.744868:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334610432/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019281.744876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019281.744879:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019281.744880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019281.744883:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3548 00000100:00080000:2.0:1734019281.744895:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019281.744900:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019281.744902:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019281.744904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019281.744912:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334610560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019281.744925:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019281/real 1734019281] req@ffff8800a78a2680 x1818247334610048/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019281.744927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019281.744930:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019281.744931:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019281.744935:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3548 00000100:00080000:2.0:1734019281.744937:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019281.744940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019281.744944:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334610048/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019281.744944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019281.744952:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334610688/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019281.744954:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019281.744957:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019281.744960:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019281.744962:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334610048:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019281.744963:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019281.744964:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019281.744968:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3548 00000100:00100000:1.0:1734019281.744970:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334610176:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019281.744970:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019281.744973:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019281.744977:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019281.744985:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334610816/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019281.744987:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019281.744995:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019281/real 1734019281] req@ffff8800a78a1880 x1818247334610176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019281.745006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334610176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019281.745014:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019281.745016:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019281.745020:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334610176:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019281.745043:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334610304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019281.745057:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019281.745063:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019281/real 1734019281] req@ffff8800a78a3b80 x1818247334610304/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019281.745081:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334610304/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019281.745089:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019281.745091:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019281.745094:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334610304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019281.745100:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334610432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019281.745112:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019281.745118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019281/real 1734019281] req@ffff8800a78a0700 x1818247334610432/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019281.745127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334610432/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019281.745134:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019281.745137:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019281.745140:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334610432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019281.745145:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334610560:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019281.745157:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019281.745163:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019281/real 1734019281] req@ffff8800a78a3480 x1818247334610560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019281.745173:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334610560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019281.745180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019281.745183:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019281.745186:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334610560:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019281.745192:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334610688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019281.745204:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019281.745210:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019281/real 1734019281] req@ffff8800a78a1f80 x1818247334610688/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019281.745219:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334610688/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019281.745226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019281.745228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019281.745231:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334610688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019281.745236:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334610816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019281.745248:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019281.745254:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019281/real 1734019281] req@ffff8800a78a0a80 x1818247334610816/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019281.745264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334610816/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019281.745271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019281.745273:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019281.745276:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334610816:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019281.904770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019282.906767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019283.908748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019284.910762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019285.912740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019286.752597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019286.752608:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019286.752613:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019286.752623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3553 00000100:00080000:2.0:1734019286.752632:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019286.752640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019286.752717:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334610944/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019286.752750:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019286.752754:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019286.752756:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019286.752761:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3553 00000100:00080000:2.0:1734019286.752764:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019286.752768:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019286.752778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019286.752789:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334611072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019286.752799:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019286.752803:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019286.752804:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019286.752808:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3553 00000100:00080000:2.0:1734019286.752811:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019286.752814:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019286.752819:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019286.752828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334611200/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019286.752836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019286.752840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019286.752842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019286.752845:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3553 00000100:00080000:2.0:1734019286.752848:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019286.752851:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019286.752854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019286.752862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334611328/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019286.752871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019286.752875:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734019286.752876:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334610944:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019286.752876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019286.752880:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3553 00000100:00080000:2.0:1734019286.752882:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019286.752885:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019286.752889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019286.752896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334611456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019286.752904:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019286.752907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019286.752908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019286.752912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3553 00000100:00080000:2.0:1734019286.752914:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019286.752917:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019286.752922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019286.752929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334611584/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019286.752937:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019286.752940:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734019286.752942:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019286.752942:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019286.752945:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3553 00000100:00080000:2.0:1734019286.752948:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019286.752951:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019286.752955:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019286.752962:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334611712/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019286.752964:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019286/real 1734019286] req@ffff8800a78a0a80 x1818247334610944/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019286.752983:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334610944/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019286.752994:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019286.752997:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019286.753002:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334610944:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019286.753009:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334611072:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019286.753026:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019286.753033:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019286/real 1734019286] req@ffff8800a78a1f80 x1818247334611072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019286.753044:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334611072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019286.753052:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019286.753055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019286.753058:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334611072:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019286.753064:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334611200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019286.753078:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019286.753084:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019286/real 1734019286] req@ffff8800a78a3480 x1818247334611200/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019286.753096:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334611200/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019286.753104:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019286.753106:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019286.753109:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334611200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019286.753134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334611328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019286.753148:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019286.753154:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019286/real 1734019286] req@ffff8800a78a0700 x1818247334611328/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019286.753164:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334611328/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019286.753171:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019286.753174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019286.753177:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334611328:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019286.753183:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334611456:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019286.753195:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019286.753201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019286/real 1734019286] req@ffff8800a78a3b80 x1818247334611456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019286.753211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334611456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019286.753219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019286.753222:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019286.753225:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334611456:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019286.753230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334611584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019286.753242:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019286.753248:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019286/real 1734019286] req@ffff8800a78a1880 x1818247334611584/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019286.753258:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334611584/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019286.753265:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019286.753267:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019286.753270:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334611584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019286.753275:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334611712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019286.753287:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019286.753293:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019286/real 1734019286] req@ffff8800a78a2680 x1818247334611712/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019286.753303:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334611712/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019286.753310:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019286.753312:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019286.753315:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334611712:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019286.914718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019287.916819:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019288.918784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019289.920758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019290.922751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019291.760636:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019291.760650:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019291.760656:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019291.760670:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3558 00000100:00080000:2.0:1734019291.760681:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019291.760691:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019291.760743:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334611840/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019291.760782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019291.760789:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019291.760791:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019291.760796:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3558 00000100:00080000:2.0:1734019291.760800:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019291.760803:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019291.760814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019291.760826:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334611968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019291.760837:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019291.760840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019291.760842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019291.760846:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3558 00000100:00080000:2.0:1734019291.760849:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019291.760853:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019291.760858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019291.760867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334612096/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019291.760877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019291.760880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019291.760882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019291.760886:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3558 00000100:00100000:1.0:1734019291.760888:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334611840:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019291.760888:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019291.760892:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019291.760897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019291.760907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334612224/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019291.760916:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019291.760919:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019291.760920:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019291.760924:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3558 00000100:00080000:2.0:1734019291.760926:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019291.760929:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019291.760934:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019291.760942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334612352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019291.760949:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019291.760953:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019291.760954:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019291.760958:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3558 00000100:00080000:2.0:1734019291.760961:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019291.760965:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019291.760969:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019291.760977:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334612480/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019291.760984:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019291.760988:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019291.760990:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019291.760991:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334611968:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019291.760994:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3558 00000100:00080000:2.0:1734019291.760997:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019291.761000:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019291.761005:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019291.761013:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334612608/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019291.761023:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334612096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019291.761041:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334612224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019291.761057:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334612352:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019291.761074:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334612480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019291.761092:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334612608:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019291.924764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019292.926725:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019293.928771:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019294.766777:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019294.766824:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019294.766842:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019294.766875:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019294.766883:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019294.766891:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019294.766898:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019294.766904:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019294.766910:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734019294.766990:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019291/real 1734019294] req@ffff8800a78a0a80 x1818247334612608/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019294.767019:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334612608/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019294.767035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019294.767041:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019294.767049:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334612608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019294.767060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019291/real 1734019294] req@ffff8800a78a3b80 x1818247334612096/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019294.767077:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334612096/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019294.767098:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019294.767103:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019294.767108:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334612096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019294.767118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019291/real 1734019294] req@ffff8800a78a0700 x1818247334612224/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019294.767135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334612224/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019294.767147:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019294.767151:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019294.767157:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334612224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019294.767167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019291/real 1734019294] req@ffff8800a78a3480 x1818247334612352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019294.767184:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334612352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019294.767197:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019294.767201:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019294.767207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334612352:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019294.767217:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019291/real 1734019294] req@ffff8800a78a1f80 x1818247334612480/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019294.767235:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334612480/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019294.767257:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019294.767260:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019294.767266:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334612480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019294.767275:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019291/real 1734019294] req@ffff8800a78a2680 x1818247334611840/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019294.767293:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334611840/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019294.767306:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019294.767309:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019294.767314:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334611840:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019294.767324:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019291/real 1734019294] req@ffff8800a78a1880 x1818247334611968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019294.767342:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334611968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019294.767355:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019294.767358:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019294.767364:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334611968:192.168.201.116@tcp:38:kworker.0 00010000:00010000:2.0:1734019294.930736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019295.932750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019296.768606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019296.768620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019296.768625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019296.768640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3563 00000100:00080000:2.0:1734019296.768652:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019296.768662:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019296.768713:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334612736/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019296.768753:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019296.768758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019296.768760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019296.768765:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3563 00000100:00080000:2.0:1734019296.768768:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019296.768772:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019296.768784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019296.768797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334612864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019296.768811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019296.768816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019296.768819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019296.768825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3563 00000100:00080000:2.0:1734019296.768830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019296.768836:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019296.768844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019296.768861:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334612992/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019296.768878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019296.768884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019296.768886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019296.768893:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334612736:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019296.768893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3563 00000100:00080000:2.0:1734019296.768898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019296.768903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019296.768911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019296.768926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334613120/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019296.768941:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019296.768946:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019296.768949:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019296.768955:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3563 00000100:00080000:2.0:1734019296.768961:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019296.768965:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019296.768967:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019296.768974:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019296.768987:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019296/real 1734019296] req@ffff8800a78a1880 x1818247334612736/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019296.768989:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334613248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019296.769003:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019296.769008:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334612736/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019296.769008:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019296.769010:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019296.769016:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3563 00000100:00080000:1.0:1734019296.769020:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019296.769021:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734019296.769024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019296.769027:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019296.769029:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334612736:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019296.769033:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019296.769038:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334612864:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019296.769044:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334613376/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019296.769053:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734019296.769056:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019296.769057:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019296.769059:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019296.769063:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3563 00000100:00100000:1.0:1734019296.769064:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019296/real 1734019296] req@ffff8800a78a2680 x1818247334612864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019296.769067:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019296.769070:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019296.769074:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019296.769076:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334612864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019296.769083:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334613504/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019296.769084:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019296.769087:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019296.769090:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334612864:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019296.769117:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334612992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019296.769132:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019296.769139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019296/real 1734019296] req@ffff8800a78a1f80 x1818247334612992/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019296.769149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334612992/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019296.769156:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019296.769160:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019296.769163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334612992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019296.769170:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334613120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019296.769183:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019296.769189:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019296/real 1734019296] req@ffff8800a78a3480 x1818247334613120/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019296.769199:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334613120/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019296.769207:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019296.769209:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019296.769212:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334613120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019296.769218:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334613248:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019296.769230:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019296.769236:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019296/real 1734019296] req@ffff8800a78a0700 x1818247334613248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019296.769246:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334613248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019296.769254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019296.769256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019296.769259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334613248:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019296.769265:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334613376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019296.769281:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019296.769287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019296/real 1734019296] req@ffff8800a78a3b80 x1818247334613376/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019296.769298:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334613376/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019296.769305:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019296.769307:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019296.769310:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334613376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019296.769316:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334613504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019296.769329:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019296.769335:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019296/real 1734019296] req@ffff8800a78a0a80 x1818247334613504/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019296.769344:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334613504/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019296.769351:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019296.769354:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019296.769357:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334613504:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019296.934716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019297.936753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019298.938740:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019299.940728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019300.942702:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019301.776600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019301.776610:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019301.776615:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019301.776626:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3568 00000100:00080000:2.0:1734019301.776636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019301.776644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019301.776683:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334613632/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019301.776718:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019301.776722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019301.776724:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019301.776728:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3568 00000100:00080000:2.0:1734019301.776732:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019301.776735:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019301.776744:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019301.776756:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334613760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019301.776765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019301.776768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019301.776770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019301.776774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3568 00000100:00080000:2.0:1734019301.776777:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019301.776781:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019301.776785:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019301.776794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334613888/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019301.776803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019301.776806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019301.776807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019301.776810:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3568 00000100:00080000:2.0:1734019301.776813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019301.776816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019301.776820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019301.776828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334614016/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019301.776837:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334613632:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019301.776837:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019301.776840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019301.776841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019301.776845:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3568 00000100:00080000:2.0:1734019301.776847:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019301.776850:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019301.776854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019301.776861:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334614144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019301.776875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019301.776878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019301.776880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019301.776883:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3568 00000100:00080000:2.0:1734019301.776886:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019301.776889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019301.776892:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019301.776900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334614272/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019301.776904:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019301.776908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019301.776911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019301.776912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019301.776915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3568 00000100:00080000:2.0:1734019301.776920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019301.776923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019301.776925:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019301/real 1734019301] req@ffff8800a78a0a80 x1818247334613632/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019301.776927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019301.776934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334614400/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019301.776945:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334613632/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019301.776956:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019301.776959:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019301.776964:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334613632:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019301.776971:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334613760:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019301.776989:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019301.776995:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019301/real 1734019301] req@ffff8800a78a3b80 x1818247334613760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019301.777006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334613760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019301.777014:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019301.777017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019301.777020:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334613760:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019301.777026:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334613888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019301.777039:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019301.777045:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019301/real 1734019301] req@ffff8800a78a0700 x1818247334613888/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019301.777055:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334613888/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019301.777063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019301.777066:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019301.777069:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334613888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019301.777094:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334614016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019301.777107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019301.777113:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019301/real 1734019301] req@ffff8800a78a3480 x1818247334614016/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019301.777124:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334614016/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019301.777131:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019301.777133:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019301.777136:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334614016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019301.777142:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334614144:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019301.777154:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019301.777160:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019301/real 1734019301] req@ffff8800a78a1f80 x1818247334614144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019301.777170:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334614144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019301.777177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019301.777180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019301.777182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334614144:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019301.777188:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334614272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019301.777204:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019301.777210:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019301/real 1734019301] req@ffff8800a78a2680 x1818247334614272/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019301.777220:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334614272/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019301.777227:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019301.777229:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019301.777232:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334614272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019301.777238:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334614400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019301.777250:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019301.777256:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019301/real 1734019301] req@ffff8800a78a1880 x1818247334614400/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019301.777265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334614400/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019301.777272:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019301.777274:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019301.777277:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334614400:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019301.944790:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019302.946742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019303.948736:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019304.950754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019305.952792:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019306.784607:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019306.784618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019306.784624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019306.784638:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3573 00000100:00080000:2.0:1734019306.784650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019306.784659:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019306.784710:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334614528/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019306.784749:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019306.784754:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019306.784756:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019306.784761:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3573 00000100:00080000:2.0:1734019306.784764:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019306.784768:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019306.784779:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019306.784792:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334614656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019306.784802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019306.784805:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019306.784807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019306.784811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3573 00000100:00080000:2.0:1734019306.784814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019306.784818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019306.784823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019306.784832:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334614784/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019306.784841:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019306.784845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019306.784847:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019306.784851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3573 00000100:00080000:2.0:1734019306.784854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019306.784857:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019306.784862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019306.784870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334614912/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019306.784880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019306.784883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019306.784884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019306.784886:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334614528:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019306.784888:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3573 00000100:00080000:2.0:1734019306.784891:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019306.784894:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019306.784898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019306.784907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334615040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019306.784915:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019306.784918:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019306.784919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019306.784923:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3573 00000100:00080000:2.0:1734019306.784926:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019306.784930:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019306.784935:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019306.784943:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334615168/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019306.784952:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019306.784955:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019306.784957:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734019306.784961:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019306.784961:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3573 00000100:00080000:2.0:1734019306.784964:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019306.784967:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019306.784972:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019306.784979:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334615296/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019306.784985:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019306/real 1734019306] req@ffff8800a78a1880 x1818247334614528/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019306.785006:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334614528/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019306.785018:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019306.785022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019306.785027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334614528:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019306.785036:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334614656:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019306.785054:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019306.785061:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019306/real 1734019306] req@ffff8800a78a2680 x1818247334614656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019306.785072:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334614656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019306.785080:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019306.785083:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019306.785086:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334614656:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019306.785093:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334614784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019306.785107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019306.785113:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019306/real 1734019306] req@ffff8800a78a1f80 x1818247334614784/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019306.785123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334614784/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019306.785138:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019306.785141:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019306.785144:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334614784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019306.785170:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334614912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019306.785184:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019306.785190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019306/real 1734019306] req@ffff8800a78a3480 x1818247334614912/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019306.785201:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334614912/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019306.785209:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019306.785211:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019306.785214:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334614912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019306.785220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334615040:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019306.785233:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019306.785239:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019306/real 1734019306] req@ffff8800a78a0700 x1818247334615040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019306.785249:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334615040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019306.785256:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019306.785259:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019306.785262:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334615040:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019306.785269:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334615168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019306.785282:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019306.785288:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019306/real 1734019306] req@ffff8800a78a3b80 x1818247334615168/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019306.785298:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334615168/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019306.785349:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019306.785351:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019306.785355:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334615168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019306.785362:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334615296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019306.785378:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019306.785386:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019306/real 1734019306] req@ffff8800a78a0a80 x1818247334615296/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019306.785397:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334615296/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019306.785404:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019306.785407:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019306.785410:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334615296:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019306.954670:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019307.956689:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019308.958750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019309.960746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019310.962737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019311.792605:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019311.792616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019311.792621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019311.792631:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3578 00000100:00080000:2.0:1734019311.792640:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019311.792647:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019311.792685:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334615424/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019311.792720:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019311.792727:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019311.792729:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019311.792733:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3578 00000100:00080000:2.0:1734019311.792737:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019311.792740:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019311.792750:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019311.792761:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334615552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019311.792770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019311.792774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019311.792775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019311.792779:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3578 00000100:00080000:2.0:1734019311.792782:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019311.792786:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019311.792790:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019311.792799:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334615680/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019311.792807:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019311.792810:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019311.792812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019311.792815:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3578 00000100:00080000:2.0:1734019311.792817:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019311.792820:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019311.792825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019311.792833:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334615424:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019311.792834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334615808/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019311.792843:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019311.792846:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019311.792848:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019311.792851:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3578 00000100:00080000:2.0:1734019311.792853:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019311.792856:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019311.792860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019311.792868:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334615936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019311.792875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019311.792878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019311.792880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019311.792883:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3578 00000100:00080000:2.0:1734019311.792895:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019311.792899:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019311.792899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019311.792903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019311.792911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334616064/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019311.792919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019311.792921:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019311/real 1734019311] req@ffff8800a78a0a80 x1818247334615424/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019311.792922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019311.792924:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019311.792928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3578 00000100:00080000:2.0:1734019311.792931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019311.792934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019311.792938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019311.792940:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334615424/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019311.792946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334616192/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019311.792951:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019311.792954:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019311.792959:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334615424:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019311.792966:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334615552:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019311.792983:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019311.792990:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019311/real 1734019311] req@ffff8800a78a3b80 x1818247334615552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019311.793001:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334615552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019311.793009:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019311.793011:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019311.793015:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334615552:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019311.793021:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334615680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019311.793034:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019311.793040:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019311/real 1734019311] req@ffff8800a78a0700 x1818247334615680/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019311.793050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334615680/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019311.793065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019311.793067:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019311.793071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334615680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019311.793095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334615808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019311.793107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019311.793114:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019311/real 1734019311] req@ffff8800a78a3480 x1818247334615808/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019311.793124:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334615808/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019311.793132:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019311.793134:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019311.793137:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334615808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019311.793142:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334615936:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019311.793154:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019311.793160:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019311/real 1734019311] req@ffff8800a78a1f80 x1818247334615936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019311.793170:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334615936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019311.793177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019311.793179:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019311.793182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334615936:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019311.793189:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334616064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019311.793201:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019311.793207:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019311/real 1734019311] req@ffff8800a78a2680 x1818247334616064/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019311.793253:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334616064/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019311.793261:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019311.793264:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019311.793267:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334616064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019311.793274:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334616192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019311.793291:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019311.793298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019311/real 1734019311] req@ffff8800a78a1880 x1818247334616192/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019311.793309:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334616192/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019311.793317:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019311.793319:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019311.793322:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334616192:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019311.964710:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019312.966695:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019313.968744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019314.970758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019315.972721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019316.800575:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019316.800586:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019316.800590:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019316.800600:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3583 00000100:00080000:2.0:1734019316.800609:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019316.800616:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019316.800653:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334616320/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019316.800686:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019316.800691:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019316.800693:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019316.800697:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3583 00000100:00080000:2.0:1734019316.800700:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019316.800703:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019316.800713:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019316.800724:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334616448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019316.800735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019316.800738:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019316.800740:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019316.800743:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3583 00000100:00080000:2.0:1734019316.800746:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019316.800749:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019316.800753:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019316.800762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334616576/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019316.800770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019316.800773:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019316.800775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019316.800778:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3583 00000100:00080000:2.0:1734019316.800780:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019316.800784:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019316.800788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019316.800795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334616704/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019316.800801:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334616320:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019316.800804:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019316.800807:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019316.800809:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019316.800812:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3583 00000100:00080000:2.0:1734019316.800815:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019316.800818:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019316.800822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019316.800830:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334616832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019316.800838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019316.800841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019316.800842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019316.800845:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3583 00000100:00080000:2.0:1734019316.800848:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019316.800851:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019316.800855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019316.800862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334616960/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019316.800867:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019316.800870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019316.800873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019316.800874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019316.800878:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3583 00000100:00100000:1.0:1734019316.800888:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019316/real 1734019316] req@ffff8800a78a1880 x1818247334616320/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019316.800890:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019316.800894:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019316.800898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019316.800906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334617088/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019316.800908:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334616320/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019316.800918:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019316.800922:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019316.800926:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334616320:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019316.800934:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334616448:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019316.800951:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019316.800958:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019316/real 1734019316] req@ffff8800a78a2680 x1818247334616448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019316.800969:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334616448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019316.800977:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019316.800979:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019316.800983:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334616448:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019316.800988:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334616576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019316.801002:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019316.801008:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019316/real 1734019316] req@ffff8800a78a1f80 x1818247334616576/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019316.801018:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334616576/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019316.801026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019316.801029:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019316.801032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334616576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019316.801056:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334616704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019316.801069:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019316.801075:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019316/real 1734019316] req@ffff8800a78a3480 x1818247334616704/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019316.801085:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334616704/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019316.801093:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019316.801095:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019316.801098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334616704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019316.801103:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334616832:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019316.801115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019316.801121:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019316/real 1734019316] req@ffff8800a78a0700 x1818247334616832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019316.801131:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334616832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019316.801170:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019316.801172:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019316.801176:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334616832:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019316.801183:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334616960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019316.801199:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019316.801206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019316/real 1734019316] req@ffff8800a78a3b80 x1818247334616960/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019316.801217:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334616960/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019316.801224:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019316.801226:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019316.801229:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334616960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019316.801235:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334617088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019316.801247:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019316.801253:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019316/real 1734019316] req@ffff8800a78a0a80 x1818247334617088/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019316.801262:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334617088/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019316.801270:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019316.801272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019316.801275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334617088:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019316.974749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019317.976750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019318.978734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019319.980763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019320.982678:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019321.808615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019321.808626:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019321.808630:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019321.808641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3588 00000100:00080000:2.0:1734019321.808650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019321.808657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019321.808695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334617216/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019321.808729:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019321.808733:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019321.808735:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019321.808740:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3588 00000100:00080000:2.0:1734019321.808743:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019321.808747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019321.808756:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019321.808767:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334617344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019321.808777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019321.808780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019321.808782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019321.808785:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3588 00000100:00080000:2.0:1734019321.808789:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019321.808792:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019321.808797:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019321.808806:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334617472/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019321.808814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019321.808817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019321.808819:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019321.808822:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3588 00000100:00080000:2.0:1734019321.808825:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019321.808828:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019321.808831:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019321.808838:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334617216:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019321.808840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334617600/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019321.808849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019321.808852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019321.808853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019321.808856:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3588 00000100:00080000:2.0:1734019321.808859:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019321.808862:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019321.808866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019321.808873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334617728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019321.808888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019321.808891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019321.808893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019321.808896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3588 00000100:00080000:2.0:1734019321.808899:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019321.808903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019321.808904:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019321.808907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019321.808914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334617856/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019321.808922:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019321.808925:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019321.808926:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019321.808927:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019321/real 1734019321] req@ffff8800a78a0a80 x1818247334617216/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019321.808930:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3588 00000100:00080000:2.0:1734019321.808932:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019321.808936:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019321.808940:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019321.808945:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334617216/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019321.808947:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334617984/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019321.808956:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019321.808960:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019321.808965:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334617216:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019321.808972:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334617344:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019321.808989:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019321.808996:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019321/real 1734019321] req@ffff8800a78a3b80 x1818247334617344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019321.809007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334617344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019321.809015:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019321.809017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019321.809021:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334617344:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019321.809026:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334617472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019321.809040:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019321.809046:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019321/real 1734019321] req@ffff8800a78a0700 x1818247334617472/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019321.809113:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334617472/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019321.809128:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019321.809131:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019321.809134:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334617472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019321.809159:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334617600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019321.809179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019321.809187:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019321/real 1734019321] req@ffff8800a78a3480 x1818247334617600/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019321.809197:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334617600/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019321.809205:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019321.809207:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019321.809210:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334617600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019321.809216:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334617728:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019321.809228:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019321.809234:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019321/real 1734019321] req@ffff8800a78a1f80 x1818247334617728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019321.809244:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334617728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019321.809251:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019321.809253:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019321.809257:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334617728:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019321.809263:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334617856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019321.809275:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019321.809281:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019321/real 1734019321] req@ffff8800a78a2680 x1818247334617856/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019321.809291:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334617856/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019321.809298:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019321.809300:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019321.809303:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334617856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019321.809309:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334617984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019321.809321:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019321.809327:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019321/real 1734019321] req@ffff8800a78a1880 x1818247334617984/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019321.809337:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334617984/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019321.809344:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019321.809346:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019321.809349:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334617984:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019321.984711:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019322.986698:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019323.988722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019324.990695:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019325.992624:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019326.816591:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019326.816603:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019326.816609:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019326.816623:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3593 00000100:00080000:2.0:1734019326.816635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019326.816645:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019326.816695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334618112/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019326.816736:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019326.816740:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019326.816742:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019326.816747:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3593 00000100:00080000:2.0:1734019326.816750:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019326.816754:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019326.816766:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019326.816778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334618240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019326.816788:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019326.816791:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019326.816793:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019326.816798:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3593 00000100:00080000:2.0:1734019326.816801:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019326.816805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019326.816809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019326.816819:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334618368/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019326.816828:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019326.816833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019326.816834:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019326.816838:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3593 00000100:00080000:2.0:1734019326.816841:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019326.816845:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019326.816849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019326.816859:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334618496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019326.816864:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334618112:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019326.816868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019326.816871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019326.816873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019326.816876:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3593 00000100:00080000:2.0:1734019326.816879:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019326.816882:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019326.816886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019326.816895:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334618624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019326.816903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019326.816906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019326.816907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019326.816911:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3593 00000100:00080000:2.0:1734019326.816913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019326.816917:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019326.816922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019326.816931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334618752/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019326.816936:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019326.816940:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019326.816943:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019326.816944:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019326.816948:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3593 00000100:00100000:1.0:1734019326.816960:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019326/real 1734019326] req@ffff8800a78a1880 x1818247334618112/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019326.816961:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019326.816995:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019326.817000:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019326.817009:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334618880/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019326.817020:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334618112/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019326.817031:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019326.817035:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019326.817040:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334618112:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019326.817048:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334618240:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019326.817069:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019326.817077:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019326/real 1734019326] req@ffff8800a78a2680 x1818247334618240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019326.817088:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334618240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019326.817097:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019326.817099:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019326.817103:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334618240:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019326.817109:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334618368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019326.817123:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019326.817130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019326/real 1734019326] req@ffff8800a78a1f80 x1818247334618368/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019326.817140:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334618368/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019326.817148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019326.817151:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019326.817154:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334618368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019326.817181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334618496:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019326.817194:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019326.817200:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019326/real 1734019326] req@ffff8800a78a3480 x1818247334618496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019326.817210:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334618496/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019326.817218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019326.817221:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019326.817224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334618496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019326.817230:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334618624:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019326.817243:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019326.817249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019326/real 1734019326] req@ffff8800a78a0700 x1818247334618624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019326.817259:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334618624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019326.817266:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019326.817269:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019326.817272:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334618624:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019326.817279:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334618752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019326.817292:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019326.817298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019326/real 1734019326] req@ffff8800a78a3b80 x1818247334618752/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019326.817308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334618752/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019326.817315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019326.817318:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019326.817321:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334618752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019326.817327:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334618880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019326.817339:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019326.817345:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019326/real 1734019326] req@ffff8800a78a0a80 x1818247334618880/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019326.817355:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334618880/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019326.817362:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019326.817364:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019326.817367:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334618880:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019326.994749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019327.996699:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019328.998714:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019330.000752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019331.002718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019331.824591:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019331.824601:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019331.824606:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019331.824616:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3598 00000100:00080000:2.0:1734019331.824625:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019331.824632:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019331.824668:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334619008/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019331.824702:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019331.824709:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019331.824711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019331.824716:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3598 00000100:00080000:2.0:1734019331.824719:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019331.824723:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019331.824732:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019331.824743:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334619136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019331.824753:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019331.824756:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019331.824758:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019331.824761:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3598 00000100:00080000:2.0:1734019331.824764:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019331.824767:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019331.824772:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019331.824780:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334619264/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019331.824789:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019331.824792:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019331.824794:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019331.824797:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3598 00000100:00080000:2.0:1734019331.824799:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019331.824803:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019331.824807:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019331.824816:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334619392/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019331.824819:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334619008:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019331.824825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019331.824828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019331.824830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019331.824833:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3598 00000100:00080000:2.0:1734019331.824835:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019331.824838:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019331.824842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019331.824850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334619520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019331.824858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019331.824860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019331.824862:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019331.824865:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3598 00000100:00080000:2.0:1734019331.824868:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019331.824871:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019331.824875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019331.824934:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019331.824959:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019331/real 1734019331] req@ffff8800a78a0a80 x1818247334619008/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019331.824970:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334619648/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019331.824979:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334619008/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019331.824980:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019331.824983:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019331.824986:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734019331.824989:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019331.824989:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3598 00000100:00080000:1.0:1734019331.824993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019331.824993:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019331.824998:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334619008:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019331.824998:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019331.825003:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019331.825006:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334619136:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019331.825011:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334619776/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019331.825028:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019331.825034:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019331/real 1734019331] req@ffff8800a78a3b80 x1818247334619136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019331.825046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334619136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019331.825054:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019331.825056:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019331.825060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334619136:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019331.825065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334619264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019331.825078:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019331.825084:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019331/real 1734019331] req@ffff8800a78a0700 x1818247334619264/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019331.825094:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334619264/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019331.825108:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019331.825111:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019331.825114:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334619264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019331.825138:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334619392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019331.825151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019331.825157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019331/real 1734019331] req@ffff8800a78a3480 x1818247334619392/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019331.825167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334619392/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019331.825175:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019331.825178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019331.825181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334619392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019331.825186:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334619520:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019331.825198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019331.825204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019331/real 1734019331] req@ffff8800a78a1f80 x1818247334619520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019331.825214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334619520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019331.825221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019331.825223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019331.825226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334619520:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019331.825233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334619648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019331.825244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019331.825251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019331/real 1734019331] req@ffff8800a78a2680 x1818247334619648/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019331.825261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334619648/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019331.825268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019331.825270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019331.825273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334619648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019331.825278:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334619776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019331.825290:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019331.825295:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019331/real 1734019331] req@ffff8800a78a1880 x1818247334619776/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019331.825305:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334619776/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019331.825312:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019331.825314:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019331.825317:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334619776:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019332.004742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019333.006720:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019334.008705:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019335.010777:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019336.012738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019336.832588:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019336.832598:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019336.832602:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019336.832613:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3603 00000100:00080000:2.0:1734019336.832622:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019336.832630:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019336.832667:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334619904/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019336.832700:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019336.832705:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019336.832707:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019336.832711:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3603 00000100:00080000:2.0:1734019336.832714:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019336.832718:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019336.832727:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019336.832739:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334620032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019336.832750:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019336.832753:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019336.832754:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019336.832758:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3603 00000100:00080000:2.0:1734019336.832760:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019336.832764:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019336.832768:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019336.832776:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334620160/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019336.832785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019336.832788:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019336.832790:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019336.832824:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3603 00000100:00080000:2.0:1734019336.832828:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019336.832831:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019336.832835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019336.832845:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334619904:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019336.832845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334620288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019336.832854:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019336.832857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019336.832859:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019336.832862:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3603 00000100:00080000:2.0:1734019336.832865:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019336.832868:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019336.832873:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019336.832880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334620416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019336.832888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019336.832891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019336.832893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019336.832896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3603 00000100:00080000:2.0:1734019336.832899:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019336.832902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019336.832906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019336.832909:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019336.832913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334620544/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019336.832921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019336.832924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019336.832926:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019336.832930:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3603 00000100:00100000:1.0:1734019336.832931:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019336/real 1734019336] req@ffff8800a78a1880 x1818247334619904/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019336.832933:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019336.832936:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019336.832940:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019336.832947:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334620672/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019336.832951:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334619904/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019336.832961:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019336.832964:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019336.832969:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334619904:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019336.832976:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334620032:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019336.832992:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019336.832999:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019336/real 1734019336] req@ffff8800a78a2680 x1818247334620032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019336.833010:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334620032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019336.833018:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019336.833020:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019336.833024:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334620032:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019336.833029:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334620160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019336.833042:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019336.833048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019336/real 1734019336] req@ffff8800a78a1f80 x1818247334620160/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019336.833059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334620160/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019336.833067:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019336.833070:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019336.833073:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334620160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019336.833097:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334620288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019336.833110:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019336.833116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019336/real 1734019336] req@ffff8800a78a3480 x1818247334620288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019336.833126:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334620288/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019336.833134:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019336.833136:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019336.833139:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334620288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019336.833145:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334620416:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019336.833157:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019336.833162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019336/real 1734019336] req@ffff8800a78a0700 x1818247334620416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019336.833172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334620416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019336.833179:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019336.833181:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019336.833185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334620416:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019336.833191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334620544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019336.833203:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019336.833209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019336/real 1734019336] req@ffff8800a78a3b80 x1818247334620544/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019336.833219:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334620544/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019336.833226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019336.833228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019336.833231:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334620544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019336.833236:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334620672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019336.833248:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019336.833254:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019336/real 1734019336] req@ffff8800a78a0a80 x1818247334620672/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019336.833264:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334620672/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019336.833271:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019336.833273:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019336.833276:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334620672:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019337.014724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019338.016748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019339.018702:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019340.020801:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019341.022782:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019341.840614:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019341.840624:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019341.840629:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019341.840639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3608 00000100:00080000:2.0:1734019341.840648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019341.840655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019341.840692:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334620800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019341.840762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019341.840767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019341.840769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019341.840774:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3608 00000100:00080000:2.0:1734019341.840777:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019341.840781:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019341.840791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019341.840804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334620928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019341.840815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019341.840818:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019341.840820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019341.840823:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3608 00000100:00080000:2.0:1734019341.840826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019341.840830:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019341.840835:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019341.840844:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334621056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019341.840852:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019341.840855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019341.840857:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019341.840860:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3608 00000100:00080000:2.0:1734019341.840862:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019341.840865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019341.840869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019341.840877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334621184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019341.840878:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334620800:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019341.840886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019341.840889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019341.840890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019341.840894:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3608 00000100:00080000:2.0:1734019341.840896:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019341.840900:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019341.840904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019341.840911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334621312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019341.840925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019341.840928:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019341.840930:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019341.840933:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3608 00000100:00080000:2.0:1734019341.840936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019341.840940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019341.840942:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019341.840943:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019341.840951:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334621440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019341.840959:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019341.840962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019341.840963:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019341.840965:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019341/real 1734019341] req@ffff8800a78a0a80 x1818247334620800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019341.840966:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3608 00000100:00080000:2.0:1734019341.840969:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019341.840972:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019341.840976:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019341.840983:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334621568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019341.840984:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334620800/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019341.840995:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019341.840999:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019341.841003:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334620800:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019341.841011:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334620928:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019341.841027:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019341.841034:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019341/real 1734019341] req@ffff8800a78a3b80 x1818247334620928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019341.841045:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334620928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019341.841053:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019341.841056:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019341.841059:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334620928:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019341.841065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334621056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019341.841079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019341.841085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019341/real 1734019341] req@ffff8800a78a0700 x1818247334621056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019341.841095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334621056/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019341.841109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019341.841112:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019341.841115:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334621056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019341.841140:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334621184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019341.841153:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019341.841159:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019341/real 1734019341] req@ffff8800a78a3480 x1818247334621184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019341.841169:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334621184/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019341.841177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019341.841179:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019341.841182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334621184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019341.841188:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334621312:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019341.841200:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019341.841206:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019341/real 1734019341] req@ffff8800a78a1f80 x1818247334621312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019341.841216:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334621312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019341.841223:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019341.841225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019341.841228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334621312:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019341.841234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334621440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019341.841246:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019341.841252:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019341/real 1734019341] req@ffff8800a78a2680 x1818247334621440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019341.841262:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334621440/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019341.841269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019341.841272:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019341.841275:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334621440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019341.841280:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334621568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019341.841293:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019341.841299:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019341/real 1734019341] req@ffff8800a78a1880 x1818247334621568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019341.841308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334621568/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019341.841315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019341.841317:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019341.841320:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334621568:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019342.024782:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019343.026754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019344.028735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019345.030771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019346.032719:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019346.848610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019346.848619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019346.848657:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019346.848669:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3613 00000100:00080000:2.0:1734019346.848678:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019346.848685:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019346.848723:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334621696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019346.848756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019346.848760:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019346.848762:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019346.848766:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3613 00000100:00080000:2.0:1734019346.848769:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019346.848773:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019346.848782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019346.848793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334621824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019346.848803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019346.848806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019346.848808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019346.848812:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3613 00000100:00080000:2.0:1734019346.848815:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019346.848818:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019346.848823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019346.848831:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334621952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019346.848840:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019346.848843:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019346.848845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019346.848848:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3613 00000100:00080000:2.0:1734019346.848851:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019346.848854:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019346.848858:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019346.848863:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334621696:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019346.848866:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334622080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019346.848875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019346.848878:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019346.848880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019346.848883:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3613 00000100:00080000:2.0:1734019346.848885:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019346.848888:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019346.848892:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019346.848899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334622208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019346.848907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019346.848910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019346.848912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019346.848915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3613 00000100:00080000:2.0:1734019346.848918:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019346.848921:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019346.848925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019346.848927:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019346.848933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334622336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019346.848941:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019346.848944:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019346.848946:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019346.848949:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019346/real 1734019346] req@ffff8800a78a1880 x1818247334621696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019346.848949:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3613 00000100:00080000:2.0:1734019346.848952:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019346.848955:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019346.848959:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019346.848966:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334622464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019346.848968:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334621696/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019346.848979:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019346.848982:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019346.848987:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334621696:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019346.848994:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334621824:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019346.849011:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019346.849018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019346/real 1734019346] req@ffff8800a78a2680 x1818247334621824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019346.849030:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334621824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019346.849037:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019346.849040:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019346.849043:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334621824:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019346.849049:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334621952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019346.849062:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019346.849068:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019346/real 1734019346] req@ffff8800a78a1f80 x1818247334621952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019346.849078:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334621952/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019346.849086:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019346.849089:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019346.849092:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334621952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019346.849116:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334622080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019346.849129:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019346.849135:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019346/real 1734019346] req@ffff8800a78a3480 x1818247334622080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019346.849145:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334622080/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019346.849153:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019346.849155:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019346.849158:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334622080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019346.849164:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334622208:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019346.849176:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019346.849182:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019346/real 1734019346] req@ffff8800a78a0700 x1818247334622208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019346.849191:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334622208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019346.849199:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019346.849201:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019346.849204:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334622208:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019346.849210:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334622336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019346.849222:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019346.849228:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019346/real 1734019346] req@ffff8800a78a3b80 x1818247334622336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019346.849238:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334622336/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019346.849246:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019346.849248:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019346.849251:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334622336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019346.849256:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334622464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019346.849268:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019346.849274:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019346/real 1734019346] req@ffff8800a78a0a80 x1818247334622464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019346.849284:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334622464/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019346.849291:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019346.849293:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019346.849296:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334622464:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019347.034736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019348.036774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019349.038770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019350.040779:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019351.042746:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019351.856638:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019351.856647:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019351.856652:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019351.856662:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3618 00000100:00080000:2.0:1734019351.856670:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019351.856678:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019351.856716:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334622592/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019351.856751:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019351.856758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019351.856760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019351.856764:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3618 00000100:00080000:2.0:1734019351.856767:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019351.856771:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019351.856780:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019351.856792:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334622720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019351.856808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019351.856812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019351.856813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019351.856817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3618 00000100:00080000:2.0:1734019351.856820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019351.856824:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019351.856828:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019351.856837:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334622848/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019351.856846:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019351.856849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019351.856851:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019351.856855:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3618 00000100:00080000:2.0:1734019351.856857:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019351.856860:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019351.856865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019351.856873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334622976/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019351.856875:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334622592:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019351.856883:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019351.856885:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019351.856887:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019351.856890:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3618 00000100:00080000:2.0:1734019351.856893:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019351.856896:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019351.856900:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019351.856907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334623104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019351.856915:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019351.856918:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019351.856920:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019351.856923:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3618 00000100:00080000:2.0:1734019351.856926:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019351.856929:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019351.856933:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019351.856941:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334623232/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019351.856942:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019351.856949:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019351.856952:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019351.856954:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019351.856957:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3618 00000100:00080000:2.0:1734019351.856960:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019351.856963:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019351.856964:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019351/real 1734019351] req@ffff8800a78a0a80 x1818247334622592/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019351.856967:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019351.856974:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334623360/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019351.856983:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334622592/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019351.856994:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019351.856998:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019351.857003:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334622592:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019351.857010:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334622720:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019351.857028:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019351.857035:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019351/real 1734019351] req@ffff8800a78a3b80 x1818247334622720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019351.857046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334622720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019351.857054:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019351.857056:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019351.857060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334622720:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019351.857066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334622848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019351.857079:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019351.857085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019351/real 1734019351] req@ffff8800a78a0700 x1818247334622848/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019351.857095:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334622848/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019351.857109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019351.857112:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019351.857115:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334622848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019351.857139:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334622976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019351.857152:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019351.857158:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019351/real 1734019351] req@ffff8800a78a3480 x1818247334622976/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019351.857168:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334622976/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019351.857176:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019351.857178:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019351.857181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334622976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019351.857187:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334623104:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019351.857199:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019351.857205:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019351/real 1734019351] req@ffff8800a78a1f80 x1818247334623104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019351.857215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334623104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019351.857222:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019351.857224:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019351.857227:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334623104:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019351.857234:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334623232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019351.857246:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019351.857252:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019351/real 1734019351] req@ffff8800a78a2680 x1818247334623232/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019351.857262:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334623232/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019351.857269:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019351.857271:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019351.857274:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334623232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019351.857279:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334623360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019351.857291:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019351.857297:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019351/real 1734019351] req@ffff8800a78a1880 x1818247334623360/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019351.857307:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334623360/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019351.857314:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019351.857316:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019351.857319:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334623360:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019352.044735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019353.046723:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019354.048674:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019355.050728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019356.052764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019356.864596:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019356.864606:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019356.864611:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019356.864622:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3623 00000100:00080000:2.0:1734019356.864631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019356.864638:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019356.864676:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334623488/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019356.864709:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019356.864713:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019356.864715:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019356.864719:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3623 00000100:00080000:2.0:1734019356.864722:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019356.864726:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019356.864735:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019356.864746:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334623616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019356.864756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019356.864760:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019356.864761:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019356.864765:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3623 00000100:00080000:2.0:1734019356.864767:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019356.864771:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019356.864775:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019356.864783:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334623744/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019356.864792:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019356.864795:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019356.864796:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019356.864800:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3623 00000100:00080000:2.0:1734019356.864802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019356.864805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019356.864809:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019356.864815:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334623488:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019356.864818:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334623872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019356.864827:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019356.864830:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019356.864831:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019356.864834:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3623 00000100:00080000:2.0:1734019356.864837:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019356.864840:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019356.864844:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019356.864852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334624000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019356.864860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019356.864863:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019356.864864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019356.864867:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3623 00000100:00080000:2.0:1734019356.864870:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019356.864873:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019356.864877:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019356.864884:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334624128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019356.864892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019356.864895:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019356.864896:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019356.864899:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3623 00000100:00080000:2.0:1734019356.864902:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019356.864905:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019356.864906:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334623616:192.168.201.116@tcp:38:kworker.0 00000008:00100000:2.0:1734019356.864909:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019356.864916:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334624256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019356.864926:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334623744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019356.864949:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334623872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019356.864965:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334624000:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019356.864979:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334624128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019356.864993:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334624256:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019357.054707:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019358.056781:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019359.058754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019359.870734:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019359.870785:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019359.870804:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019359.870838:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019359.870846:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019359.870853:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019359.870860:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019359.870867:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019359.870873:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734019359.870891:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019356/real 1734019359] req@ffff8800a78a3480 x1818247334623872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019359.870920:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334623872/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019359.870933:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019359.870937:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019359.870942:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334623872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019359.870950:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019356/real 1734019359] req@ffff8800a78a0700 x1818247334624000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019359.870961:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334624000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019359.870969:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019359.870972:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019359.870977:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334624000:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019359.870984:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019356/real 1734019359] req@ffff8800a78a3b80 x1818247334624128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019359.870995:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334624128/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019359.871002:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019359.871005:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019359.871008:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334624128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019359.871014:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019356/real 1734019359] req@ffff8800a78a0a80 x1818247334624256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019359.871024:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334624256/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019359.871032:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019359.871034:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019359.871037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334624256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019359.871043:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019356/real 1734019359] req@ffff8800a78a1880 x1818247334623488/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019359.871054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334623488/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019359.871062:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019359.871064:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019359.871067:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334623488:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019359.871074:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019356/real 1734019359] req@ffff8800a78a2680 x1818247334623616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019359.871085:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334623616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019359.871092:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019359.871095:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019359.871098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334623616:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019359.871104:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019356/real 1734019359] req@ffff8800a78a1f80 x1818247334623744/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019359.871122:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334623744/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019359.871133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019359.871137:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019359.871142:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334623744:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019360.060726:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019361.062755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019361.872617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019361.872627:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019361.872632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019361.872641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3628 00000100:00080000:2.0:1734019361.872650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019361.872658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019361.872694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334624384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019361.872728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019361.872732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019361.872734:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019361.872738:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3628 00000100:00080000:2.0:1734019361.872741:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019361.872745:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019361.872754:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019361.872765:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334624512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019361.872775:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019361.872778:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019361.872780:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019361.872784:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3628 00000100:00080000:2.0:1734019361.872787:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019361.872791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019361.872795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019361.872804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334624640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019361.872812:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019361.872815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019361.872817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019361.872820:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3628 00000100:00080000:2.0:1734019361.872822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019361.872825:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019361.872829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019361.872836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334624768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019361.872845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019361.872848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019361.872850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019361.872851:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334624384:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019361.872853:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3628 00000100:00080000:2.0:1734019361.872855:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019361.872858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019361.872862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019361.872870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334624896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019361.872884:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019361.872888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019361.872889:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019361.872893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3628 00000100:00080000:2.0:1734019361.872895:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019361.872898:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019361.872902:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019361.872909:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334625024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019361.872917:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734019361.872918:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019361.872920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019361.872921:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019361.872925:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3628 00000100:00080000:2.0:1734019361.872927:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019361.872931:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019361.872934:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019361.872938:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019361/real 1734019361] req@ffff8800a78a1f80 x1818247334624384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019361.872942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334625152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019361.872958:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334624384/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019361.872969:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019361.872973:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019361.872978:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334624384:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019361.872985:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334624512:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019361.873003:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019361.873010:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019361/real 1734019361] req@ffff8800a78a2680 x1818247334624512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019361.873020:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334624512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019361.873029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019361.873032:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019361.873035:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334624512:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019361.873041:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334624640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019361.873054:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019361.873060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019361/real 1734019361] req@ffff8800a78a1880 x1818247334624640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019361.873070:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334624640/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019361.873078:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019361.873080:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019361.873083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334624640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019361.873107:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334624768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019361.873124:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019361.873130:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019361/real 1734019361] req@ffff8800a78a0a80 x1818247334624768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019361.873141:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334624768/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019361.873148:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019361.873151:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019361.873153:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334624768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019361.873159:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334624896:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019361.873171:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019361.873177:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019361/real 1734019361] req@ffff8800a78a3b80 x1818247334624896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019361.873187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334624896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019361.873194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019361.873196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019361.873199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334624896:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019361.873205:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334625024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019361.873217:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019361.873223:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019361/real 1734019361] req@ffff8800a78a0700 x1818247334625024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019361.873240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334625024/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019361.873247:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019361.873249:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019361.873252:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334625024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019361.873257:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334625152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019361.873269:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019361.873275:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019361/real 1734019361] req@ffff8800a78a3480 x1818247334625152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019361.873285:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334625152/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019361.873292:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019361.873294:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019361.873297:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334625152:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019362.064780:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019363.066751:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019364.068783:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019365.070780:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019366.072795:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019366.880565:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019366.880575:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019366.880581:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019366.880591:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3633 00000100:00080000:2.0:1734019366.880601:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019366.880608:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019366.880644:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334625280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019366.880678:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019366.880682:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019366.880684:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019366.880689:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3633 00000100:00080000:2.0:1734019366.880692:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019366.880696:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019366.880705:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019366.880716:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334625408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019366.880726:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019366.880729:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019366.880731:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019366.880735:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3633 00000100:00080000:2.0:1734019366.880738:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019366.880742:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019366.880746:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019366.880755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334625536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019366.880763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019366.880767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019366.880769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019366.880772:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3633 00000100:00080000:2.0:1734019366.880775:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019366.880778:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019366.880782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019366.880789:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334625664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019366.880799:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334625280:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019366.880799:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019366.880802:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019366.880803:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019366.880807:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3633 00000100:00080000:2.0:1734019366.880809:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019366.880812:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019366.880816:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019366.880823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334625792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019366.880831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019366.880834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019366.880836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019366.880839:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3633 00000100:00080000:2.0:1734019366.880842:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019366.880846:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019366.880850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019366.880858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334625920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019366.880864:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019366.880866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019366.880869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019366.880870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019366.880874:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3633 00000100:00080000:2.0:1734019366.880876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019366.880879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019366.880883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019366.880887:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019366/real 1734019366] req@ffff8800a78a3480 x1818247334625280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019366.880891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334626048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019366.880906:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334625280/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019366.880917:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019366.880920:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019366.880925:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334625280:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019366.880932:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334625408:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019366.880949:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019366.880956:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019366/real 1734019366] req@ffff8800a78a0700 x1818247334625408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019366.880968:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334625408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019366.880976:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019366.880978:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019366.880982:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334625408:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019366.880988:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334625536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019366.881001:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019366.881008:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019366/real 1734019366] req@ffff8800a78a3b80 x1818247334625536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019366.881018:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334625536/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019366.881025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019366.881027:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019366.881031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334625536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019366.881056:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334625664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019366.881069:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019366.881075:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019366/real 1734019366] req@ffff8800a78a0a80 x1818247334625664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019366.881086:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334625664/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019366.881093:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019366.881095:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019366.881098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334625664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019366.881104:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334625792:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019366.881116:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019366.881122:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019366/real 1734019366] req@ffff8800a78a1880 x1818247334625792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019366.881132:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334625792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019366.881139:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019366.881141:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019366.881144:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334625792:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019366.881150:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334625920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019366.881161:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019366.881167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019366/real 1734019366] req@ffff8800a78a2680 x1818247334625920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019366.881178:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334625920/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019366.881186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019366.881188:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019366.881191:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334625920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019366.881196:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334626048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019366.881208:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019366.881214:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019366/real 1734019366] req@ffff8800a78a1f80 x1818247334626048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019366.881224:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334626048/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019366.881231:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019366.881233:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019366.881236:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334626048:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019367.074752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019368.076722:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019369.078752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019370.080764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019371.082738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019371.888583:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019371.888593:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019371.888597:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019371.888607:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3638 00000100:00080000:2.0:1734019371.888616:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019371.888623:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019371.888659:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334626176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019371.888693:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019371.888700:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019371.888702:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019371.888706:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3638 00000100:00080000:2.0:1734019371.888710:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019371.888714:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019371.888723:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019371.888734:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334626304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019371.888744:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019371.888747:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019371.888749:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019371.888753:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3638 00000100:00080000:2.0:1734019371.888755:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019371.888759:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019371.888764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019371.888772:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334626432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019371.888781:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019371.888784:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019371.888785:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019371.888788:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3638 00000100:00080000:2.0:1734019371.888791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019371.888794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019371.888798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019371.888804:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334626176:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019371.888806:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334626560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019371.888815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019371.888818:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019371.888820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019371.888823:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3638 00000100:00080000:2.0:1734019371.888826:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019371.888829:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019371.888833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019371.888840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334626688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019371.888847:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019371.888850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019371.888852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019371.888855:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3638 00000100:00080000:2.0:1734019371.888858:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019371.888861:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019371.888865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019371.888868:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019371.888872:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334626816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019371.888880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019371.888884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019371.888885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019371.888889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3638 00000100:00100000:1.0:1734019371.888890:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019371/real 1734019371] req@ffff8800a78a1f80 x1818247334626176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019371.888892:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019371.888895:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019371.888899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019371.888906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334626944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019371.888909:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334626176/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019371.888920:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019371.888923:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019371.888928:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334626176:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019371.888935:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334626304:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019371.888952:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019371.888958:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019371/real 1734019371] req@ffff8800a78a2680 x1818247334626304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019371.888970:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334626304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019371.888978:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019371.888980:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019371.888984:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334626304:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019371.888990:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334626432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019371.889003:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019371.889009:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019371/real 1734019371] req@ffff8800a78a1880 x1818247334626432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019371.889019:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334626432/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019371.889026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019371.889028:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019371.889032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334626432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019371.889058:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334626560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019371.889071:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019371.889078:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019371/real 1734019371] req@ffff8800a78a0a80 x1818247334626560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019371.889088:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334626560/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019371.889096:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019371.889098:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019371.889101:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334626560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019371.889107:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334626688:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019371.889119:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019371.889125:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019371/real 1734019371] req@ffff8800a78a3b80 x1818247334626688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019371.889135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334626688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019371.889142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019371.889145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019371.889148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334626688:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019371.889153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334626816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019371.889165:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019371.889171:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019371/real 1734019371] req@ffff8800a78a0700 x1818247334626816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019371.889182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334626816/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019371.889189:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019371.889191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019371.889194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334626816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019371.889241:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334626944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019371.889258:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019371.889266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019371/real 1734019371] req@ffff8800a78a3480 x1818247334626944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019371.889276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334626944/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019371.889284:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019371.889286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019371.889289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334626944:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019372.084716:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019373.086728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019374.088805:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019375.090773:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019376.092771:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019376.896607:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019376.896617:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019376.896622:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019376.896632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3643 00000100:00080000:2.0:1734019376.896641:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019376.896648:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019376.896684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334627072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019376.896717:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019376.896721:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019376.896723:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019376.896728:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3643 00000100:00080000:2.0:1734019376.896731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019376.896734:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019376.896744:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019376.896755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334627200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019376.896777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019376.896778:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019376.896779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019376.896781:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3643 00000100:00080000:2.0:1734019376.896782:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019376.896784:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019376.896787:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019376.896792:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334627328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019376.896797:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019376.896798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019376.896799:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019376.896801:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3643 00000100:00080000:2.0:1734019376.896802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019376.896804:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019376.896806:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019376.896807:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334627072:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019376.896811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334627456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019376.896815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019376.896817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019376.896818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019376.896819:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3643 00000100:00080000:2.0:1734019376.896821:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019376.896839:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019376.896843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019376.896851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334627584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019376.896860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019376.896863:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019376.896864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019376.896867:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3643 00000100:00080000:2.0:1734019376.896870:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019376.896874:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019376.896877:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019376.896878:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019376.896885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334627712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019376.896893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019376.896896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019376.896897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019376.896900:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019376/real 1734019376] req@ffff8800a78a3480 x1818247334627072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019376.896901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3643 00000100:00080000:2.0:1734019376.896903:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019376.896906:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019376.896910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019376.896918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334627840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019376.896919:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334627072/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019376.896930:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019376.896933:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019376.896938:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334627072:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019376.896946:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334627200:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019376.896963:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019376.896969:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019376/real 1734019376] req@ffff8800a78a0700 x1818247334627200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019376.896981:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334627200/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019376.896989:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019376.896991:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019376.896995:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334627200:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019376.897001:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334627328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019376.897014:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019376.897020:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019376/real 1734019376] req@ffff8800a78a3b80 x1818247334627328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019376.897030:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334627328/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019376.897037:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019376.897040:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019376.897043:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334627328:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019376.897069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334627456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019376.897083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019376.897090:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019376/real 1734019376] req@ffff8800a78a0a80 x1818247334627456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019376.897100:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334627456/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019376.897108:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019376.897110:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019376.897159:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334627456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019376.897165:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334627584:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019376.897182:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019376.897190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019376/real 1734019376] req@ffff8800a78a1880 x1818247334627584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019376.897201:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334627584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019376.897210:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019376.897212:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019376.897215:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334627584:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019376.897220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334627712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019376.897232:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019376.897238:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019376/real 1734019376] req@ffff8800a78a2680 x1818247334627712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019376.897250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334627712/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019376.897257:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019376.897259:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019376.897262:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334627712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019376.897268:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334627840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019376.897280:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019376.897286:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019376/real 1734019376] req@ffff8800a78a1f80 x1818247334627840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019376.897296:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334627840/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019376.897303:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019376.897305:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019376.897308:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334627840:192.168.201.116@tcp:8:kworker.0 00010000:00010000:2.0:1734019377.094734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019378.096743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019379.098706:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019380.100755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019381.102774:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019381.904588:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019381.904597:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019381.904602:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019381.904612:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3648 00000100:00080000:2.0:1734019381.904622:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019381.904629:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019381.904667:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334627968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019381.904703:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019381.904707:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019381.904708:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019381.904713:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3648 00000100:00080000:2.0:1734019381.904716:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019381.904719:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019381.904729:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019381.904740:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334628096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019381.904750:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019381.904753:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019381.904755:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019381.904759:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3648 00000100:00080000:2.0:1734019381.904762:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019381.904766:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019381.904770:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019381.904779:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334628224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019381.904788:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019381.904791:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019381.904793:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019381.904796:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3648 00000100:00080000:2.0:1734019381.904798:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019381.904801:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019381.904805:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019381.904813:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334628352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019381.904822:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019381.904825:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019381.904826:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019381.904830:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3648 00000100:00080000:2.0:1734019381.904832:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019381.904835:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019381.904839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019381.904842:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334627968:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019381.904847:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334628480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019381.904861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019381.904864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019381.904866:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019381.904869:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3648 00000100:00080000:2.0:1734019381.904871:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019381.904874:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019381.904878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019381.904886:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334628608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019381.904893:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019381.904896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019381.904898:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019381.904901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3648 00000100:00080000:2.0:1734019381.904903:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019381.904906:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019381.904910:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019381.904917:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334628736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019381.904936:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019381.904966:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019381/real 1734019381] req@ffff8800a78a1f80 x1818247334627968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019381.904989:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334627968/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019381.905002:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019381.905006:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019381.905012:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334627968:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019381.905020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334628096:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019381.905074:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019381.905082:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019381/real 1734019381] req@ffff8800a78a2680 x1818247334628096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019381.905094:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334628096/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019381.905103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019381.905106:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019381.905110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334628096:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019381.905116:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334628224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019381.905132:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019381.905138:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019381/real 1734019381] req@ffff8800a78a1880 x1818247334628224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019381.905149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334628224/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019381.905157:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019381.905159:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019381.905163:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334628224:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019381.905195:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334628352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019381.905210:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019381.905216:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019381/real 1734019381] req@ffff8800a78a0a80 x1818247334628352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019381.905227:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334628352/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019381.905235:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019381.905238:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019381.905241:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334628352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019381.905247:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334628480:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019381.905260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019381.905266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019381/real 1734019381] req@ffff8800a78a3b80 x1818247334628480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019381.905276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334628480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019381.905283:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019381.905286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019381.905288:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334628480:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019381.905294:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334628608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019381.905307:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019381.905313:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019381/real 1734019381] req@ffff8800a78a0700 x1818247334628608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019381.905324:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334628608/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019381.905331:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019381.905333:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019381.905336:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334628608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019381.905342:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334628736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019381.905356:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019381.905362:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019381/real 1734019381] req@ffff8800a78a3480 x1818247334628736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019381.905372:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334628736/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019381.905379:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019381.905381:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019381.905384:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334628736:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019382.104748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019383.106792:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019384.108765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019385.110685:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019386.112755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019386.912615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019386.912628:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019386.912635:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019386.912648:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3653 00000100:00080000:2.0:1734019386.912660:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019386.912670:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019386.912726:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334628864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019386.912765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019386.912770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019386.912772:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019386.912777:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3653 00000100:00080000:2.0:1734019386.912780:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019386.912784:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019386.912796:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019386.912809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334628992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019386.912819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019386.912822:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019386.912824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019386.912829:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3653 00000100:00080000:2.0:1734019386.912831:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019386.912835:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019386.912840:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019386.912850:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334629120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019386.912858:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019386.912862:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019386.912864:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019386.912869:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3653 00000100:00080000:2.0:1734019386.912871:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019386.912875:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019386.912879:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019386.912884:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334628864:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019386.912888:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334629248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019386.912897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019386.912900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019386.912902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019386.912905:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3653 00000100:00080000:2.0:1734019386.912908:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019386.912911:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019386.912915:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019386.912923:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334629376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019386.912931:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019386.912934:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019386.912936:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019386.912939:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3653 00000100:00080000:2.0:1734019386.912978:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019386.912982:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019386.912987:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019386.912997:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334629504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019386.913007:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019386.913010:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019386.913012:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734019386.913015:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019386.913016:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3653 00000100:00080000:2.0:1734019386.913019:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019386.913022:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019386.913026:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019386.913034:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334629632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019386.913045:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019386/real 1734019386] req@ffff8800a78a3480 x1818247334628864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019386.913066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334628864/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019386.913079:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019386.913083:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019386.913088:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334628864:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019386.913096:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334628992:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019386.913116:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019386.913123:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019386/real 1734019386] req@ffff8800a78a0700 x1818247334628992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019386.913135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334628992/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019386.913143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019386.913145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019386.913149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334628992:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019386.913154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334629120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019386.913169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019386.913176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019386/real 1734019386] req@ffff8800a78a3b80 x1818247334629120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019386.913186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334629120/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019386.913193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019386.913195:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019386.913199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334629120:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019386.913228:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334629248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019386.913242:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019386.913249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019386/real 1734019386] req@ffff8800a78a0a80 x1818247334629248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019386.913260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334629248/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019386.913267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019386.913269:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019386.913272:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334629248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019386.913278:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334629376:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019386.913290:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019386.913297:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019386/real 1734019386] req@ffff8800a78a1880 x1818247334629376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019386.913306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334629376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019386.913313:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019386.913315:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019386.913318:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334629376:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019386.913324:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334629504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019386.913337:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019386.913343:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019386/real 1734019386] req@ffff8800a78a2680 x1818247334629504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019386.913355:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334629504/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019386.913363:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019386.913365:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019386.913368:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334629504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019386.913373:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334629632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019386.913386:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019386.913392:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019386/real 1734019386] req@ffff8800a78a1f80 x1818247334629632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019386.913401:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334629632/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019386.913408:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019386.913410:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019386.913413:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334629632:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019387.114755:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019388.116706:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019389.118803:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019390.120768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019391.122749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019391.920638:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019391.920651:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019391.920658:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019391.920672:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3658 00000100:00080000:2.0:1734019391.920684:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019391.920694:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019391.920747:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334629760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019391.920785:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019391.920789:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019391.920791:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019391.920796:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3658 00000100:00080000:2.0:1734019391.920799:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019391.920803:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019391.920816:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019391.920829:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334629888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019391.920839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019391.920843:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019391.920845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019391.920849:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3658 00000100:00080000:2.0:1734019391.920852:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019391.920885:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019391.920891:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019391.920902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334630016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019391.920912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019391.920915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019391.920917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019391.920920:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3658 00000100:00080000:2.0:1734019391.920923:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019391.920927:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019391.920928:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334629760:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019391.920931:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019391.920940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334630144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019391.920949:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019391.920952:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019391.920954:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019391.920958:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3658 00000100:00080000:2.0:1734019391.920961:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019391.920964:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019391.920969:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019391.920977:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334630272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019391.920985:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019391.920988:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019391.920989:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019391.920993:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3658 00000100:00080000:2.0:1734019391.920996:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019391.920999:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019391.921003:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019391.921011:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334630400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019391.921016:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019391.921019:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019391.921023:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019391.921024:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019391.921028:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3658 00000100:00080000:2.0:1734019391.921031:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019391.921034:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019391.921038:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019391.921042:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019391/real 1734019391] req@ffff8800a78a1f80 x1818247334629760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019391.921047:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334630528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019391.921064:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334629760/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019391.921077:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019391.921081:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019391.921086:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334629760:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019391.921093:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334629888:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019391.921111:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019391.921118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019391/real 1734019391] req@ffff8800a78a2680 x1818247334629888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019391.921129:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334629888/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019391.921138:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019391.921140:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019391.921144:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334629888:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019391.921170:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334630016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019391.921185:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019391.921192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019391/real 1734019391] req@ffff8800a78a1880 x1818247334630016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019391.921202:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334630016/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019391.921211:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019391.921213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019391.921216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334630016:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019391.921223:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334630144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019391.921236:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019391.921242:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019391/real 1734019391] req@ffff8800a78a0a80 x1818247334630144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019391.921252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334630144/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019391.921259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019391.921261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019391.921264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334630144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019391.921270:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334630272:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019391.921283:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019391.921289:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019391/real 1734019391] req@ffff8800a78a3b80 x1818247334630272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019391.921299:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334630272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019391.921306:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019391.921308:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019391.921312:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334630272:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019391.921317:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334630400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019391.921331:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019391.921337:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019391/real 1734019391] req@ffff8800a78a0700 x1818247334630400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019391.921348:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334630400/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019391.921355:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019391.921357:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019391.921360:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334630400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019391.921365:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334630528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019391.921378:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019391.921384:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019391/real 1734019391] req@ffff8800a78a3480 x1818247334630528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019391.921393:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334630528/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019391.921400:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019391.921402:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019391.921405:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334630528:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019392.124735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019393.126716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019394.128750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019395.130725:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019396.132743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019396.928624:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019396.928637:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019396.928643:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019396.928656:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3663 00000100:00080000:2.0:1734019396.928669:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019396.928679:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019396.928731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334630656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019396.928770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019396.928814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019396.928816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019396.928821:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3663 00000100:00080000:2.0:1734019396.928825:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019396.928829:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019396.928842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019396.928856:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334630784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019396.928867:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019396.928871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019396.928873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019396.928877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3663 00000100:00080000:2.0:1734019396.928880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019396.928883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019396.928890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1734019396.928899:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334630656:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019396.928901:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334630912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019396.928912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019396.928916:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019396.928917:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019396.928921:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3663 00000100:00080000:2.0:1734019396.928924:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019396.928928:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019396.928932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019396.928940:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334631040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019396.928949:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019396.928952:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019396.928953:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019396.928957:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3663 00000100:00080000:2.0:1734019396.928963:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019396.928966:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019396.928970:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734019396.928973:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019396.928979:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334631168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019396.928987:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019396.928990:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019396.928992:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019396.928996:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3663 00000100:00100000:1.0:1734019396.928997:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019396/real 1734019396] req@ffff8800a78a3480 x1818247334630656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019396.929000:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019396.929003:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019396.929008:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019396.929016:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334631296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019396.929017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334630656/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019396.929024:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019396.929028:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019396.929029:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734019396.929030:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019396.929033:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019396.929033:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3663 00000100:00080000:2.0:1734019396.929035:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019396.929039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334630656:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019396.929039:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019396.929043:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019396.929048:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334630784:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019396.929052:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334631424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019396.929066:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019396.929074:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019396/real 1734019396] req@ffff8800a78a0700 x1818247334630784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019396.929085:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334630784/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019396.929094:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019396.929097:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019396.929100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334630784:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019396.929126:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334630912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019396.929141:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019396.929147:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019396/real 1734019396] req@ffff8800a78a3b80 x1818247334630912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019396.929158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334630912/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019396.929166:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019396.929168:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019396.929171:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334630912:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019396.929179:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334631040:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019396.929192:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019396.929198:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019396/real 1734019396] req@ffff8800a78a0a80 x1818247334631040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019396.929209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334631040/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019396.929216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019396.929218:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019396.929221:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334631040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019396.929227:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334631168:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019396.929240:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019396.929246:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019396/real 1734019396] req@ffff8800a78a1880 x1818247334631168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019396.929257:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334631168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019396.929265:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019396.929267:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019396.929270:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334631168:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019396.929276:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334631296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019396.929289:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019396.929295:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019396/real 1734019396] req@ffff8800a78a2680 x1818247334631296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019396.929306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334631296/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019396.929313:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019396.929315:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019396.929318:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334631296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019396.929324:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334631424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019396.929336:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019396.929342:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019396/real 1734019396] req@ffff8800a78a1f80 x1818247334631424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019396.929352:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334631424/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019396.929359:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019396.929361:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019396.929364:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334631424:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019397.134818:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019398.136741:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019399.138805:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019400.140764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019401.142775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019401.936638:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019401.936651:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019401.936657:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019401.936672:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3668 00000100:00080000:2.0:1734019401.936683:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019401.936731:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019401.936784:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334631552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019401.936833:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019401.936837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019401.936839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019401.936844:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3668 00000100:00080000:2.0:1734019401.936848:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019401.936852:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019401.936864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019401.936877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334631680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019401.936886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019401.936890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019401.936892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019401.936896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3668 00000100:00080000:2.0:1734019401.936898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019401.936902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019401.936908:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019401.936918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334631808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019401.936927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019401.936930:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019401.936932:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019401.936935:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3668 00000100:00080000:2.0:1734019401.936938:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019401.936942:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019401.936946:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019401.936953:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334631552:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019401.936956:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334631936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019401.936966:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019401.936969:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019401.936971:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019401.936975:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3668 00000100:00080000:2.0:1734019401.936977:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019401.936981:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019401.936985:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019401.936993:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334632064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019401.937001:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019401.937004:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019401.937006:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019401.937009:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3668 00000100:00080000:2.0:1734019401.937014:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019401.937018:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019401.937022:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019401.937025:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019401.937031:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334632192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019401.937040:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019401.937044:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019401.937046:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019401.937048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019401/real 1734019401] req@ffff8800a78a1f80 x1818247334631552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019401.937049:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3668 00000100:00080000:2.0:1734019401.937053:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019401.937057:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019401.937061:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019401.937068:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334631552/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019401.937069:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334632320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019401.937080:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019401.937084:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019401.937089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334631552:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019401.937098:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334631680:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019401.937115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019401.937123:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019401/real 1734019401] req@ffff8800a78a2680 x1818247334631680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019401.937134:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334631680/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019401.937142:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019401.937145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019401.937149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334631680:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019401.937155:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334631808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019401.937169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019401.937176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019401/real 1734019401] req@ffff8800a78a1880 x1818247334631808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019401.937186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334631808/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019401.937193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019401.937196:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019401.937199:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334631808:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019401.937227:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334631936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019401.937241:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019401.937247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019401/real 1734019401] req@ffff8800a78a0a80 x1818247334631936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019401.937258:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334631936/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019401.937266:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019401.937268:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019401.937271:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334631936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019401.937277:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334632064:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019401.937289:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019401.937296:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019401/real 1734019401] req@ffff8800a78a3b80 x1818247334632064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019401.937306:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334632064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019401.937313:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019401.937315:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019401.937318:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334632064:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019401.937324:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334632192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019401.937337:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019401.937343:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019401/real 1734019401] req@ffff8800a78a0700 x1818247334632192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019401.937354:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334632192/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019401.937362:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019401.937364:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019401.937367:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334632192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019401.937373:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334632320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019401.937385:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019401.937391:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019401/real 1734019401] req@ffff8800a78a3480 x1818247334632320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019401.937401:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334632320/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019401.937408:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019401.937411:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019401.937414:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334632320:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019402.144747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019403.146746:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019404.148751:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019405.150781:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019406.152780:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019406.944602:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019406.944653:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019406.944658:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019406.944669:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3673 00000100:00080000:2.0:1734019406.944677:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019406.944685:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019406.944723:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334632448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019406.944757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019406.944761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019406.944763:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019406.944767:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3673 00000100:00080000:2.0:1734019406.944770:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019406.944774:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019406.944784:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019406.944795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334632576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019406.944806:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019406.944809:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019406.944811:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019406.944814:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3673 00000100:00080000:2.0:1734019406.944817:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019406.944820:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019406.944825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019406.944833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334632704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019406.944842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019406.944845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019406.944846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019406.944850:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3673 00000100:00080000:2.0:1734019406.944852:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019406.944855:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019406.944859:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019406.944864:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334632448:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019406.944867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334632832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019406.944876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019406.944879:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019406.944881:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019406.944884:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3673 00000100:00080000:2.0:1734019406.944887:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019406.944891:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019406.944895:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019406.944902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334632960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019406.944910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019406.944913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019406.944914:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019406.944918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3673 00000100:00080000:2.0:1734019406.944920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019406.944924:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019406.944928:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019406.944930:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019406.944935:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334633088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019406.944943:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019406.944946:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019406.944947:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019406.944951:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3673 00000100:00100000:1.0:1734019406.944952:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019406/real 1734019406] req@ffff8800a78a3480 x1818247334632448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019406.944954:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019406.944957:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019406.944961:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019406.944968:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334633216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019406.944971:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334632448/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019406.944983:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019406.944986:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019406.944991:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334632448:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019406.944999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334632576:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019406.945015:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019406.945022:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019406/real 1734019406] req@ffff8800a78a0700 x1818247334632576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019406.945034:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334632576/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019406.945042:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019406.945045:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019406.945048:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334632576:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019406.945054:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334632704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019406.945068:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019406.945074:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019406/real 1734019406] req@ffff8800a78a3b80 x1818247334632704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019406.945084:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334632704/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019406.945091:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019406.945094:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019406.945097:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334632704:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019406.945122:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334632832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019406.945136:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019406.945142:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019406/real 1734019406] req@ffff8800a78a0a80 x1818247334632832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019406.945153:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334632832/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019406.945160:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019406.945162:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019406.945165:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334632832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019406.945171:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334632960:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019406.945183:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019406.945189:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019406/real 1734019406] req@ffff8800a78a1880 x1818247334632960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019406.945198:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334632960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019406.945206:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019406.945208:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019406.945211:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334632960:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019406.945217:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334633088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019406.945229:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019406.945235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019406/real 1734019406] req@ffff8800a78a2680 x1818247334633088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019406.945245:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334633088/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019406.945253:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019406.945255:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019406.945258:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334633088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019406.945263:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334633216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019406.945275:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019406.945281:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019406/real 1734019406] req@ffff8800a78a1f80 x1818247334633216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019406.945291:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334633216/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019406.945298:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019406.945300:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019406.945303:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334633216:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019407.154723:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019408.156849:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019409.158797:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019410.160769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019411.162733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019411.952618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019411.952627:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019411.952632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019411.952641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3678 00000100:00080000:2.0:1734019411.952650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019411.952658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019411.952697:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334633344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019411.952728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019411.952732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019411.952734:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019411.952739:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3678 00000100:00080000:2.0:1734019411.952742:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019411.952746:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019411.952755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019411.952767:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334633472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019411.952776:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019411.952779:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019411.952781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019411.952785:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3678 00000100:00080000:2.0:1734019411.952788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019411.952791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019411.952797:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019411.952805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334633600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019411.952814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019411.952817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019411.952818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019411.952822:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3678 00000100:00080000:2.0:1734019411.952824:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019411.952827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019411.952831:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019411.952840:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334633344:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019411.952840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334633728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019411.952848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019411.952851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019411.952853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019411.952856:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3678 00000100:00080000:2.0:1734019411.952859:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019411.952862:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019411.952866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019411.952873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334633856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019411.952882:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019411.952884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019411.952886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019411.952890:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3678 00000100:00080000:2.0:1734019411.952894:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019411.952897:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019411.952901:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019411.952904:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019411.952909:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334633984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019411.952917:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019411.952920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019411.952921:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019411.952925:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3678 00000100:00100000:1.0:1734019411.952926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019411/real 1734019411] req@ffff8800a78a1f80 x1818247334633344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019411.952927:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019411.952931:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019411.952934:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019411.952942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334634112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019411.952945:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334633344/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019411.952957:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019411.952960:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019411.952965:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334633344:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019411.952972:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334633472:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019411.952988:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019411.952995:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019411/real 1734019411] req@ffff8800a78a2680 x1818247334633472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019411.953007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334633472/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019411.953015:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019411.953017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019411.953020:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334633472:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019411.953026:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334633600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019411.953039:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019411.953045:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019411/real 1734019411] req@ffff8800a78a1880 x1818247334633600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019411.953055:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334633600/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019411.953063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019411.953065:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019411.953068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334633600:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019411.953093:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334633728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019411.953107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019411.953113:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019411/real 1734019411] req@ffff8800a78a0a80 x1818247334633728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019411.953124:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334633728/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019411.953131:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019411.953133:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019411.953136:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334633728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019411.953142:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334633856:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019411.953154:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019411.953160:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019411/real 1734019411] req@ffff8800a78a3b80 x1818247334633856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019411.953170:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334633856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019411.953177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019411.953179:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019411.953182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334633856:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019411.953187:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334633984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019411.953199:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019411.953205:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019411/real 1734019411] req@ffff8800a78a0700 x1818247334633984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019411.953215:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334633984/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019411.953223:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019411.953225:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019411.953228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334633984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019411.953233:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334634112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019411.953245:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019411.953251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019411/real 1734019411] req@ffff8800a78a3480 x1818247334634112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019411.953261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334634112/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019411.953267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019411.953269:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019411.953273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334634112:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019412.164763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019413.166754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019414.168743:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019415.170748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019416.172753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019416.960629:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019416.960639:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019416.960643:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019416.960654:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3683 00000100:00080000:2.0:1734019416.960663:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019416.960670:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019416.960708:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334634240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019416.960751:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019416.960757:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019416.960759:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019416.960766:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3683 00000100:00080000:2.0:1734019416.960771:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019416.960777:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019416.960789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019416.960808:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334634368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019416.960812:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334634240:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019416.960824:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019416.960829:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019416.960831:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019416.960836:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3683 00000100:00080000:2.0:1734019416.960840:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019416.960845:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019416.960851:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019416.960863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334634496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019416.960876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019416.960880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734019416.960883:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019416.960883:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019416.960888:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3683 00000100:00080000:2.0:1734019416.960891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019416.960896:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019416.960902:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019416.960907:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019416/real 1734019416] req@ffff8800a78a3480 x1818247334634240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019416.960914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334634624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019416.960927:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019416.960930:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334634240/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019416.960932:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019416.960935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019416.960940:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3683 00000100:00080000:2.0:1734019416.960944:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734019416.960945:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019416.960949:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734019416.960950:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000008:00100000:2.0:1734019416.960955:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019416.960956:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334634240:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019416.960966:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334634752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019416.960978:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019416.960983:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334634368:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019416.960984:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019416.960986:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019416.960992:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3683 00000100:00080000:2.0:1734019416.960995:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019416.961001:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019416.961007:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019416.961009:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019416.961020:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019416/real 1734019416] req@ffff8800a78a0700 x1818247334634368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019416.961020:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334634880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019416.961035:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019416.961038:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334634368/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019416.961040:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019416.961043:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019416.961048:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3683 00000100:00080000:1.0:1734019416.961051:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019416.961053:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734019416.961055:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019416.961058:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019416.961060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334634368:192.168.201.116@tcp:38:kworker.0 00000008:00100000:2.0:1734019416.961064:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019416.961069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334634496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019416.961075:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334635008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019416.961092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019416.961103:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019416/real 1734019416] req@ffff8800a78a3b80 x1818247334634496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019416.961121:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334634496/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019416.961133:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019416.961137:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019416.961142:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334634496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019416.961152:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334634624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019416.961179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019416.961190:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019416/real 1734019416] req@ffff8800a78a0a80 x1818247334634624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019416.961207:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334634624/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019416.961219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019416.961222:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019416.961228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334634624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019416.961236:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334634752:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019416.961257:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019416.961268:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019416/real 1734019416] req@ffff8800a78a1880 x1818247334634752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019416.961284:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334634752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019416.961295:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019416.961298:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019416.961301:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334634752:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019416.961320:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334634880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019416.961338:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019416.961345:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019416/real 1734019416] req@ffff8800a78a2680 x1818247334634880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019416.961356:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334634880/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019416.961363:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019416.961365:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019416.961368:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334634880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019416.961374:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334635008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019416.961385:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019416.961391:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019416/real 1734019416] req@ffff8800a78a1f80 x1818247334635008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019416.961401:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334635008/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019416.961408:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019416.961410:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019416.961413:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334635008:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019417.174761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019418.176750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019419.178778:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019420.180780:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019421.182730:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019421.968617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019421.968631:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019421.968637:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019421.968650:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3688 00000100:00080000:2.0:1734019421.968661:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019421.968672:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019421.968723:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334635136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019421.968762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019421.968766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019421.968769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019421.968774:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3688 00000100:00080000:2.0:1734019421.968777:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019421.968781:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019421.968792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019421.968805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334635264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019421.968816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019421.968820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019421.968822:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019421.968825:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3688 00000100:00080000:2.0:1734019421.968828:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019421.968832:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019421.968837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019421.968847:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334635392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019421.968856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019421.968859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019421.968861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019421.968865:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3688 00000100:00080000:2.0:1734019421.968867:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019421.968871:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019421.968875:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019421.968884:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334635520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019421.968892:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334635136:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019421.968894:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019421.968897:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019421.968898:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019421.968902:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3688 00000100:00080000:2.0:1734019421.968906:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019421.968909:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019421.968914:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019421.968922:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334635648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019421.968930:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019421.968933:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019421.968935:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019421.968938:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3688 00000100:00080000:2.0:1734019421.968941:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019421.968944:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019421.968948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019421.968956:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334635776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019421.968964:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019421.968968:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019421.968969:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019421.968973:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3688 00000100:00080000:2.0:1734019421.968976:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019421.968979:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019421.968983:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019421.968991:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334635904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019421.969010:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334635264:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019421.969033:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334635392:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019421.969060:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334635520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019421.969076:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334635648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019421.969092:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334635776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019421.969108:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334635904:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019422.184756:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019423.186731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019424.188762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019424.974766:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019424.974816:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019424.974836:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019424.974871:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019424.974880:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019424.974887:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019424.974893:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019424.974900:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019424.974906:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734019424.975012:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019421/real 1734019424] req@ffff8800a78a0a80 x1818247334635520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019424.975063:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334635520/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019424.975083:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019424.975090:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019424.975098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334635520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019424.975113:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019421/real 1734019424] req@ffff8800a78a3b80 x1818247334635648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019424.975133:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334635648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019424.975146:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019424.975150:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019424.975156:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334635648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019424.975167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019421/real 1734019424] req@ffff8800a78a0700 x1818247334635776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019424.975211:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334635776/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019424.975225:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019424.975230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019424.975236:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334635776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019424.975247:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019421/real 1734019424] req@ffff8800a78a3480 x1818247334635904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019424.975267:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334635904/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019424.975279:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019424.975283:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019424.975289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334635904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019424.975359:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019421/real 1734019424] req@ffff8800a78a1f80 x1818247334635136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019424.975383:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334635136/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019424.975395:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019424.975400:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019424.975405:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334635136:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019424.975418:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019421/real 1734019424] req@ffff8800a78a2680 x1818247334635264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019424.975448:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334635264/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019424.975461:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019424.975620:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019424.975628:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334635264:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019424.975640:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019421/real 1734019424] req@ffff8800a78a1880 x1818247334635392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019424.975662:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334635392/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019424.975676:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019424.975680:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019424.975684:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334635392:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019425.190716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019426.192748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019426.976600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019426.976611:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019426.976616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019426.976626:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3693 00000100:00080000:2.0:1734019426.976635:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019426.976642:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019426.976681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334636032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019426.976716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019426.976720:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019426.976722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019426.976727:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3693 00000100:00080000:2.0:1734019426.976730:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019426.976734:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019426.976743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019426.976755:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334636160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019426.976765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019426.976768:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019426.976770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019426.976773:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3693 00000100:00080000:2.0:1734019426.976776:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019426.976780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019426.976785:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019426.976794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334636288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019426.976803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019426.976806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019426.976808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019426.976811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3693 00000100:00080000:2.0:1734019426.976814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019426.976817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019426.976821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019426.976829:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334636416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019426.976838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019426.976841:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019426.976843:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019426.976846:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3693 00000100:00080000:2.0:1734019426.976848:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019426.976852:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019426.976854:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334636032:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019426.976856:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019426.976864:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334636544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019426.976873:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019426.976876:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019426.976878:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019426.976881:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3693 00000100:00080000:2.0:1734019426.976884:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019426.976887:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019426.976891:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019426.976898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334636672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019426.976906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019426.976909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019426.976911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019426.976914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3693 00000100:00080000:2.0:1734019426.976916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019426.976919:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019426.976923:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019426.976930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334636800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019426.976951:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019426.976976:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019426/real 1734019426] req@ffff8800a78a1880 x1818247334636032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019426.976998:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334636032/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019426.977010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019426.977014:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019426.977021:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334636032:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019426.977029:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334636160:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019426.977048:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019426.977056:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019426/real 1734019426] req@ffff8800a78a2680 x1818247334636160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019426.977066:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334636160/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019426.977075:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019426.977078:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019426.977082:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334636160:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019426.977088:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334636288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019426.977103:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019426.977110:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019426/real 1734019426] req@ffff8800a78a1f80 x1818247334636288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019426.977120:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334636288/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019426.977127:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019426.977130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019426.977133:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334636288:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019426.977163:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334636416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019426.977178:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019426.977185:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019426/real 1734019426] req@ffff8800a78a3480 x1818247334636416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019426.977196:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334636416/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019426.977204:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019426.977207:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019426.977210:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334636416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019426.977216:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334636544:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019426.977229:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019426.977235:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019426/real 1734019426] req@ffff8800a78a0700 x1818247334636544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019426.977245:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334636544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019426.977252:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019426.977255:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019426.977258:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334636544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019426.977264:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334636672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019426.977314:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019426.977322:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019426/real 1734019426] req@ffff8800a78a3b80 x1818247334636672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019426.977333:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334636672/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019426.977349:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019426.977352:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019426.977355:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334636672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019426.977361:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334636800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019426.977375:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019426.977381:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019426/real 1734019426] req@ffff8800a78a0a80 x1818247334636800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019426.977391:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334636800/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019426.977399:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019426.977402:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019426.977405:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334636800:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019427.194728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019428.196765:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019429.198784:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019430.200734:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019431.202744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019431.984574:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019431.984584:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019431.984588:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019431.984599:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3698 00000100:00080000:2.0:1734019431.984608:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019431.984615:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019431.984652:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334636928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019431.984695:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019431.984699:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019431.984701:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019431.984705:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3698 00000100:00080000:2.0:1734019431.984709:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019431.984713:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019431.984722:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019431.984733:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334637056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019431.984742:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019431.984746:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019431.984747:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019431.984751:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3698 00000100:00080000:2.0:1734019431.984754:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019431.984757:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019431.984762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019431.984770:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334637184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019431.984780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019431.984783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019431.984784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019431.984788:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3698 00000100:00080000:2.0:1734019431.984790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019431.984794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019431.984798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019431.984806:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334637312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019431.984816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019431.984819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019431.984820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019431.984824:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3698 00000100:00080000:2.0:1734019431.984826:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019431.984829:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019431.984833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019431.984840:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334636928:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019431.984841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334637440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019431.984849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019431.984852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019431.984853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019431.984857:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3698 00000100:00080000:2.0:1734019431.984859:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019431.984862:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019431.984866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019431.984874:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334637568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019431.984882:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019431.984885:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019431.984887:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019431.984890:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3698 00000100:00080000:2.0:1734019431.984893:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019431.984896:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019431.984899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019431.984907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334637696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019431.984939:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019431.984969:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019431/real 1734019431] req@ffff8800a78a0a80 x1818247334636928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019431.984992:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334636928/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019431.985005:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019431.985009:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019431.985016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334636928:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019431.985025:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334637056:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019431.985044:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019431.985051:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019431/real 1734019431] req@ffff8800a78a3b80 x1818247334637056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019431.985062:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334637056/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019431.985071:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019431.985074:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019431.985077:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334637056:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019431.985084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334637184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019431.985100:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019431.985106:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019431/real 1734019431] req@ffff8800a78a0700 x1818247334637184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019431.985117:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334637184/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019431.985125:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019431.985127:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019431.985130:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334637184:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019431.985161:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334637312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019431.985175:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019431.985222:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019431/real 1734019431] req@ffff8800a78a3480 x1818247334637312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019431.985235:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334637312/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019431.985244:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019431.985247:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019431.985250:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334637312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019431.985257:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334637440:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019431.985274:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019431.985281:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019431/real 1734019431] req@ffff8800a78a1f80 x1818247334637440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019431.985292:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334637440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019431.985301:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019431.985303:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019431.985306:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334637440:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019431.985312:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334637568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019431.985324:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019431.985330:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019431/real 1734019431] req@ffff8800a78a2680 x1818247334637568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019431.985340:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334637568/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019431.985355:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019431.985358:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019431.985361:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334637568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019431.985366:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334637696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019431.985380:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019431.985386:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019431/real 1734019431] req@ffff8800a78a1880 x1818247334637696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019431.985396:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334637696/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019431.985403:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019431.985406:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019431.985408:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334637696:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019432.204740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019433.206702:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019434.208763:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019435.210738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019436.212793:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019436.992612:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019436.992622:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019436.992627:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019436.992637:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3703 00000100:00080000:2.0:1734019436.992646:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019436.992655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019436.992693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334637824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019436.992727:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019436.992732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019436.992734:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019436.992738:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3703 00000100:00080000:2.0:1734019436.992741:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019436.992745:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019436.992755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019436.992766:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334637952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019436.992777:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019436.992780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019436.992782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019436.992786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3703 00000100:00080000:2.0:1734019436.992788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019436.992792:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019436.992796:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019436.992805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334638080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019436.992814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019436.992817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019436.992818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019436.992822:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3703 00000100:00080000:2.0:1734019436.992824:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019436.992827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019436.992831:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019436.992839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334638208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019436.992851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019436.992854:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019436.992856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019436.992859:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3703 00000100:00080000:2.0:1734019436.992862:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019436.992866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019436.992870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019436.992877:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334637824:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019436.992878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334638336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019436.992886:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019436.992889:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019436.992891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019436.992894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3703 00000100:00080000:2.0:1734019436.992896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019436.992900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019436.992904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019436.992911:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334638464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019436.992919:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019436.992922:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019436.992923:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019436.992926:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3703 00000100:00080000:2.0:1734019436.992929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019436.992932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019436.992936:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019436.992943:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334638592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019436.992978:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019436.993010:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019436/real 1734019436] req@ffff8800a78a1880 x1818247334637824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019436.993031:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334637824/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019436.993044:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019436.993049:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019436.993055:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334637824:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019436.993064:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334637952:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019436.993083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019436.993090:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019436/real 1734019436] req@ffff8800a78a2680 x1818247334637952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019436.993144:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334637952/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019436.993153:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019436.993157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019436.993160:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334637952:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019436.993167:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334638080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019436.993186:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019436.993193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019436/real 1734019436] req@ffff8800a78a1f80 x1818247334638080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019436.993204:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334638080/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019436.993212:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019436.993215:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019436.993218:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334638080:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019436.993248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334638208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019436.993267:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019436.993273:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019436/real 1734019436] req@ffff8800a78a3480 x1818247334638208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019436.993285:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334638208/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019436.993293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019436.993295:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019436.993298:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334638208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019436.993304:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334638336:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019436.993317:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019436.993323:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019436/real 1734019436] req@ffff8800a78a0700 x1818247334638336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019436.993333:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334638336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019436.993341:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019436.993343:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019436.993346:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334638336:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019436.993352:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334638464:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019436.993364:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019436.993370:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019436/real 1734019436] req@ffff8800a78a3b80 x1818247334638464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019436.993387:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334638464/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019436.993394:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019436.993396:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019436.993399:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334638464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019436.993405:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334638592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019436.993418:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019436.993424:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019436/real 1734019436] req@ffff8800a78a0a80 x1818247334638592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019436.993434:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334638592/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019436.993441:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019436.993444:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019436.993446:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334638592:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019437.214760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019438.216764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019439.218688:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019440.220774:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019441.222794:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019442.000627:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019442.000640:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019442.000646:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019442.000659:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3708 00000100:00080000:2.0:1734019442.000671:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019442.000680:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019442.000731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334638720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019442.000770:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019442.000774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019442.000776:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019442.000781:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3708 00000100:00080000:2.0:1734019442.000785:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019442.000789:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019442.000800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019442.000813:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334638848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019442.000823:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019442.000826:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019442.000828:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019442.000833:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3708 00000100:00080000:2.0:1734019442.000835:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019442.000840:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019442.000845:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019442.000855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334638976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019442.000865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019442.000868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019442.000870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019442.000873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3708 00000100:00080000:2.0:1734019442.000876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019442.000879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019442.000883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019442.000892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334639104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019442.000894:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334638720:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019442.000901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019442.000904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019442.000905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019442.000909:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3708 00000100:00080000:2.0:1734019442.000911:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019442.000915:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019442.000919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019442.000927:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334639232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019442.000936:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019442.000939:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019442.000941:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019442.000945:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3708 00000100:00080000:2.0:1734019442.000948:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019442.000952:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019442.000956:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019442.000964:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334639360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019442.000972:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019442.000975:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019442.000977:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734019442.000981:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019442.000981:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3708 00000100:00080000:2.0:1734019442.000984:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019442.000987:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019442.000991:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019442.000999:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334639488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019442.001008:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019442/real 1734019442] req@ffff8800a78a0a80 x1818247334638720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019442.001056:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334638720/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019442.001069:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019442.001073:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019442.001078:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334638720:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019442.001087:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334638848:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019442.001107:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019442.001115:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019442/real 1734019442] req@ffff8800a78a3b80 x1818247334638848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019442.001126:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334638848/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019442.001135:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019442.001137:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019442.001141:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334638848:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019442.001146:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334638976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019442.001161:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019442.001167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019442/real 1734019442] req@ffff8800a78a0700 x1818247334638976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019442.001177:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334638976/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019442.001185:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019442.001187:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019442.001190:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334638976:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019442.001220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334639104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019442.001235:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019442.001241:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019442/real 1734019442] req@ffff8800a78a3480 x1818247334639104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019442.001252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334639104/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019442.001260:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019442.001262:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019442.001265:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334639104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019442.001271:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334639232:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019442.001283:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019442.001289:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019442/real 1734019442] req@ffff8800a78a1f80 x1818247334639232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019442.001299:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334639232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019442.001306:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019442.001308:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019442.001311:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334639232:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019442.001317:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334639360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019442.001330:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019442.001336:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019442/real 1734019442] req@ffff8800a78a2680 x1818247334639360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019442.001347:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334639360/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019442.001355:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019442.001357:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019442.001360:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334639360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019442.001365:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334639488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019442.001378:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019442.001384:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019442/real 1734019442] req@ffff8800a78a1880 x1818247334639488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019442.001393:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334639488/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019442.001400:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019442.001402:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019442.001405:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334639488:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019442.224709:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019443.226785:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019444.228733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019445.230716:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019446.232782:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019447.008624:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019447.008635:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019447.008640:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019447.008651:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3713 00000100:00080000:2.0:1734019447.008660:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019447.008667:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019447.008705:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334639616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019447.008747:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019447.008751:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019447.008753:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019447.008757:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3713 00000100:00080000:2.0:1734019447.008761:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019447.008764:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019447.008774:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019447.008785:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334639744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019447.008795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019447.008798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019447.008799:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019447.008803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3713 00000100:00080000:2.0:1734019447.008806:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019447.008810:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019447.008814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019447.008823:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334639872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019447.008831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019447.008834:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019447.008836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019447.008840:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3713 00000100:00080000:2.0:1734019447.008842:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019447.008846:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019447.008850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019447.008858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334640000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019447.008866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019447.008869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019447.008871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019447.008874:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3713 00000100:00080000:2.0:1734019447.008876:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019447.008879:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019447.008883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019447.008890:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334640128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019447.008900:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019447.008903:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019447.008904:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019447.008908:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3713 00000100:00080000:2.0:1734019447.008910:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019447.008913:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334639616:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019447.008913:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019447.008918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019447.008925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334640256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019447.008956:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019447.008959:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019447.008961:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019447.008965:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3713 00000100:00080000:2.0:1734019447.008968:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019447.008971:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019447.008976:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019447.008984:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334640384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019447.009043:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019447.009074:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019447/real 1734019447] req@ffff8800a78a1880 x1818247334639616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019447.009097:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334639616/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019447.009111:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019447.009115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019447.009120:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334639616:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019447.009129:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334639744:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019447.009148:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019447.009155:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019447/real 1734019447] req@ffff8800a78a2680 x1818247334639744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019447.009167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334639744/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019447.009176:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019447.009179:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019447.009182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334639744:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019447.009189:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334639872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019447.009204:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019447.009210:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019447/real 1734019447] req@ffff8800a78a1f80 x1818247334639872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019447.009220:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334639872/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019447.009228:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019447.009230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019447.009234:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334639872:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019447.009241:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334640000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019447.009256:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019447.009262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019447/real 1734019447] req@ffff8800a78a3480 x1818247334640000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019447.009273:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334640000/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019447.009281:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019447.009284:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019447.009287:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334640000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019447.009319:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334640128:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019447.009333:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019447.009339:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019447/real 1734019447] req@ffff8800a78a0700 x1818247334640128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019447.009349:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334640128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019447.009358:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019447.009361:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019447.009364:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334640128:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019447.009369:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334640256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019447.009382:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019447.009388:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019447/real 1734019447] req@ffff8800a78a3b80 x1818247334640256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019447.009399:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334640256/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019447.009407:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019447.009410:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019447.009413:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334640256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019447.009418:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334640384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019447.009431:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019447.009437:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019447/real 1734019447] req@ffff8800a78a0a80 x1818247334640384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019447.009447:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334640384/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019447.009454:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019447.009456:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019447.009459:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334640384:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019447.234774:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019448.236766:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019449.238733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019450.240759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019451.242626:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019452.016600:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019452.016619:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019452.016626:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019452.016644:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3718 00000100:00080000:2.0:1734019452.016659:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019452.016685:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019452.016726:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334640512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019452.016755:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019452.016758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019452.016759:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019452.016763:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3718 00000100:00080000:2.0:1734019452.016765:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019452.016768:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019452.016777:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019452.016786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334640640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019452.016794:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019452.016796:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019452.016798:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019452.016800:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3718 00000100:00080000:2.0:1734019452.016802:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019452.016803:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334640512:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019452.016805:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019452.016808:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019452.016815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334640768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019452.016821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019452.016823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019452.016824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019452.016826:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3718 00000100:00080000:2.0:1734019452.016828:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019452.016831:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019452.016834:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019452.016840:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334640896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019452.016862:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019452.016864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019452.016866:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019452.016868:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3718 00000100:00080000:2.0:1734019452.016870:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019452.016873:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019452.016876:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019452.016883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334641024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019452.016888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019452.016890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019452.016891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019452.016894:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3718 00000100:00080000:2.0:1734019452.016896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019452.016898:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019452.016899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019452.016901:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019452.016908:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334641152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019452.016912:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019452.016915:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019452.016915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019452.016918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3718 00000100:00100000:1.0:1734019452.016920:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019452/real 1734019452] req@ffff8800a78a0a80 x1818247334640512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019452.016920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019452.016922:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019452.016925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019452.016931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334641280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019452.016935:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334640512/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019452.016944:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019452.016947:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019452.016950:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334640512:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019452.016971:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334640640:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019452.016985:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019452.016989:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019452/real 1734019452] req@ffff8800a78a3b80 x1818247334640640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019452.016997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334640640/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019452.017003:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019452.017004:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019452.017006:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334640640:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019452.017010:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334640768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019452.017020:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019452.017024:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019452/real 1734019452] req@ffff8800a78a0700 x1818247334640768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019452.017030:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334640768/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019452.017035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019452.017036:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019452.017038:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334640768:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019452.017043:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334640896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019452.017052:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019452.017056:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019452/real 1734019452] req@ffff8800a78a3480 x1818247334640896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019452.017063:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334640896/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019452.017068:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019452.017069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019452.017071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334640896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019452.017074:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334641024:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019452.017083:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019452.017086:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019452/real 1734019452] req@ffff8800a78a1f80 x1818247334641024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019452.017093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334641024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019452.017097:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019452.017098:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019452.017100:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334641024:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019452.017104:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334641152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019452.017113:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019452.017116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019452/real 1734019452] req@ffff8800a78a2680 x1818247334641152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019452.017123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334641152/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019452.017127:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019452.017128:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019452.017130:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334641152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019452.017134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334641280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019452.017142:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019452.017145:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019452/real 1734019452] req@ffff8800a78a1880 x1818247334641280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019452.017151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334641280/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019452.017155:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019452.017157:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019452.017158:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334641280:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019452.244747:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019453.246790:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019454.248753:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019455.250733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019456.252735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019457.024626:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019457.024638:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019457.024643:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019457.024657:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3723 00000100:00080000:2.0:1734019457.024668:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019457.024678:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019457.024729:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334641408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019457.024811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019457.024815:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019457.024817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019457.024823:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3723 00000100:00080000:2.0:1734019457.024827:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019457.024831:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019457.024843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019457.024858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334641536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019457.024869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019457.024872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019457.024874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019457.024875:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334641408:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019457.024878:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3723 00000100:00080000:2.0:1734019457.024881:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019457.024885:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019457.024890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019457.024901:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334641664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019457.024910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019457.024913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019457.024915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019457.024918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3723 00000100:00080000:2.0:1734019457.024921:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019457.024924:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019457.024928:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019457.024937:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334641792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019457.024945:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019457.024948:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019457.024949:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019457.024953:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3723 00000100:00080000:2.0:1734019457.024956:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019457.024960:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019457.024961:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019457.024965:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019457.024973:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334641920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019457.024981:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019457.024984:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019457.024986:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019457.024987:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019457/real 1734019457] req@ffff8800a78a1880 x1818247334641408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019457.024990:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3723 00000100:00080000:2.0:1734019457.024993:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019457.024997:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019457.025001:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019457.025009:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334641408/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019457.025009:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334642048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019457.025017:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019457.025021:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1734019457.025022:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019457.025022:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734019457.025026:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019457.025026:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3723 00000100:00080000:2.0:1734019457.025028:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019457.025031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334641408:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019457.025031:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019457.025035:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019457.025043:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334642176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019457.025058:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334641536:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019457.025077:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019457.025084:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019457/real 1734019457] req@ffff8800a78a2680 x1818247334641536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019457.025096:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334641536/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019457.025104:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019457.025107:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019457.025110:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334641536:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019457.025117:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334641664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019457.025131:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019457.025137:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019457/real 1734019457] req@ffff8800a78a1f80 x1818247334641664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019457.025147:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334641664/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019457.025154:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019457.025156:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019457.025160:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334641664:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019457.025166:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334641792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019457.025180:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019457.025186:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019457/real 1734019457] req@ffff8800a78a3480 x1818247334641792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019457.025196:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334641792/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019457.025203:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019457.025206:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019457.025208:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334641792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019457.025214:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334641920:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019457.025227:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019457.025233:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019457/real 1734019457] req@ffff8800a78a0700 x1818247334641920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019457.025243:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334641920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019457.025250:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019457.025252:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019457.025255:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334641920:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019457.025261:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334642048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019457.025274:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019457.025280:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019457/real 1734019457] req@ffff8800a78a3b80 x1818247334642048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019457.025291:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334642048/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019457.025299:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019457.025301:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019457.025304:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334642048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019457.025309:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334642176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019457.025322:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019457.025328:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019457/real 1734019457] req@ffff8800a78a0a80 x1818247334642176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019457.025338:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334642176/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019457.025345:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019457.025347:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019457.025350:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334642176:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019457.254737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019458.256798:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019459.258798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019460.260775:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019461.262802:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019462.032601:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019462.032612:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019462.032616:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019462.032628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3728 00000100:00080000:2.0:1734019462.032636:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019462.032644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019462.032740:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334642304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019462.032782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019462.032787:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019462.032789:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019462.032793:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3728 00000100:00080000:2.0:1734019462.032797:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019462.032801:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019462.032811:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019462.032825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334642432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019462.032834:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019462.032837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019462.032839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019462.032843:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3728 00000100:00080000:2.0:1734019462.032845:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019462.032849:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019462.032853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019462.032863:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334642560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019462.032872:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019462.032875:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019462.032876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019462.032880:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3728 00000100:00080000:2.0:1734019462.032882:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019462.032886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019462.032890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019462.032898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334642688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019462.032907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019462.032910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019462.032911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019462.032915:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3728 00000100:00080000:2.0:1734019462.032917:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019462.032921:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019462.032924:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334642304:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019462.032925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019462.032933:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334642816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019462.032941:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019462.032944:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019462.032945:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019462.032949:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3728 00000100:00080000:2.0:1734019462.032951:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019462.032954:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019462.032958:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019462.032966:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334642944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019462.032974:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019462.032977:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019462.032979:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019462.032982:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3728 00000100:00080000:2.0:1734019462.032985:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019462.032988:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019462.032992:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019462.032999:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334643072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019462.033020:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019462.033050:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019462/real 1734019462] req@ffff8800a78a0a80 x1818247334642304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019462.033073:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334642304/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019462.033086:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019462.033090:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019462.033096:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334642304:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019462.033105:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334642432:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019462.033123:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019462.033131:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019462/real 1734019462] req@ffff8800a78a3b80 x1818247334642432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019462.033142:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334642432/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019462.033151:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019462.033154:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019462.033157:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334642432:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019462.033163:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334642560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019462.033179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019462.033185:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019462/real 1734019462] req@ffff8800a78a0700 x1818247334642560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019462.033195:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334642560/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019462.033203:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019462.033205:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019462.033209:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334642560:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019462.033239:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334642688:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019462.033254:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019462.033260:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019462/real 1734019462] req@ffff8800a78a3480 x1818247334642688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019462.033272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334642688/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019462.033279:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019462.033282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019462.033285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334642688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019462.033291:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334642816:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019462.033304:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019462.033310:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019462/real 1734019462] req@ffff8800a78a1f80 x1818247334642816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019462.033320:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334642816/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019462.033328:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019462.033330:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019462.033333:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334642816:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019462.033339:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334642944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019462.033351:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019462.033357:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019462/real 1734019462] req@ffff8800a78a2680 x1818247334642944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019462.033369:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334642944/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019462.033377:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019462.033379:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019462.033382:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334642944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019462.033388:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334643072:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019462.033401:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019462.033407:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019462/real 1734019462] req@ffff8800a78a1880 x1818247334643072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019462.033417:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334643072/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019462.033424:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019462.033427:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019462.033430:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334643072:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019462.264754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019463.266766:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019464.268742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019465.270813:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019466.272783:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019467.040627:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019467.040640:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019467.040645:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019467.040655:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3733 00000100:00080000:2.0:1734019467.040663:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019467.040671:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019467.040709:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334643200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019467.040746:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019467.040750:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019467.040752:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019467.040756:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3733 00000100:00080000:2.0:1734019467.040759:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019467.040763:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019467.040773:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019467.040784:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334643328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019467.040795:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019467.040798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019467.040800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019467.040803:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3733 00000100:00080000:2.0:1734019467.040806:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019467.040809:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019467.040814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019467.040822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334643456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019467.040831:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019467.040835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019467.040836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019467.040840:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3733 00000100:00080000:2.0:1734019467.040842:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019467.040845:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019467.040849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019467.040858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334643584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019467.040866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019467.040869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019467.040870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019467.040873:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3733 00000100:00080000:2.0:1734019467.040876:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019467.040880:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019467.040884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019467.040893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334643712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019467.040896:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334643200:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019467.040902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019467.040905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019467.040906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019467.040910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3733 00000100:00080000:2.0:1734019467.040912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019467.040915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019467.040919:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019467.040926:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334643840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019467.040934:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019467.040937:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019467.040939:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019467.040942:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3733 00000100:00080000:2.0:1734019467.040944:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019467.040947:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019467.040951:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019467.040958:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334643968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019467.040994:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019467.041024:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019467/real 1734019467] req@ffff8800a78a1880 x1818247334643200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019467.041046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334643200/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019467.041060:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019467.041064:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019467.041070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334643200:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019467.041077:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334643328:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019467.041096:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019467.041103:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019467/real 1734019467] req@ffff8800a78a2680 x1818247334643328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019467.041114:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334643328/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019467.041122:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019467.041126:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019467.041129:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334643328:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019467.041136:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334643456:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019467.041151:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019467.041157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019467/real 1734019467] req@ffff8800a78a1f80 x1818247334643456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019467.041167:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334643456/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019467.041175:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019467.041177:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019467.041181:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334643456:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019467.041188:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334643584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019467.041202:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019467.041209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019467/real 1734019467] req@ffff8800a78a3480 x1818247334643584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019467.041219:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334643584/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019467.041227:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019467.041230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019467.041233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334643584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019467.041266:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334643712:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019467.041280:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019467.041286:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019467/real 1734019467] req@ffff8800a78a0700 x1818247334643712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019467.041297:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334643712/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019467.041304:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019467.041307:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019467.041310:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334643712:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019467.041316:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334643840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019467.041328:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019467.041334:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019467/real 1734019467] req@ffff8800a78a3b80 x1818247334643840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019467.041345:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334643840/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019467.041353:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019467.041356:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019467.041358:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334643840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019467.041364:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334643968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019467.041377:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019467.041383:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019467/real 1734019467] req@ffff8800a78a0a80 x1818247334643968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019467.041392:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334643968/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019467.041399:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019467.041402:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019467.041405:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334643968:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019467.274795:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019468.276836:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019469.278741:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019470.280827:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019471.282779:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019472.048616:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019472.048630:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019472.048635:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019472.048649:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3738 00000100:00080000:2.0:1734019472.048659:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019472.048669:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019472.048719:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334644096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019472.048760:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019472.048766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019472.048769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019472.048776:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3738 00000100:00080000:2.0:1734019472.048781:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019472.048788:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019472.048802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019472.048822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334644224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019472.048838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019472.048841:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334644096:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019472.048843:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019472.048845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019472.048852:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3738 00000100:00080000:2.0:1734019472.048856:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019472.048861:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019472.048869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019472.048883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334644352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019472.048895:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019472.048900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019472.048903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019472.048908:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3738 00000100:00080000:2.0:1734019472.048912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019472.048916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019472.048921:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000800:00000100:1.0:1734019472.048934:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019472.048934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334644480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019472.048947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019472.048952:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019472.048954:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019472.048959:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3738 00000100:00100000:1.0:1734019472.048961:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019472/real 1734019472] req@ffff8800a78a0a80 x1818247334644096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019472.048963:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019472.048969:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019472.048975:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019472.048987:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334644096/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019472.048988:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334644608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019472.049001:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734019472.049004:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019472.049007:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1734019472.049008:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019472.049009:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019472.049015:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334644096:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019472.049016:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3738 00000100:00080000:2.0:1734019472.049020:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019472.049026:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019472.049032:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019472.049044:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334644736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019472.049045:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334644224:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019472.049058:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019472.049064:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019472.049066:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019472.049072:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3738 00000800:00000100:1.0:1734019472.049073:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019472.049077:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019472.049082:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019472.049085:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019472/real 1734019472] req@ffff8800a78a3b80 x1818247334644224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019472.049090:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019472.049102:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334644224/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019472.049103:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334644864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019472.049116:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019472.049119:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019472.049124:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334644224:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019472.049134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334644352:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019472.049162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019472.049173:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019472/real 1734019472] req@ffff8800a78a0700 x1818247334644352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019472.049192:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334644352/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019472.049205:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019472.049209:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019472.049215:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334644352:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019472.049224:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334644480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019472.049246:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019472.049254:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019472/real 1734019472] req@ffff8800a78a3480 x1818247334644480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019472.049265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334644480/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019472.049273:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019472.049275:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019472.049279:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334644480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019472.049284:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334644608:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019472.049297:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019472.049303:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019472/real 1734019472] req@ffff8800a78a1f80 x1818247334644608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019472.049313:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334644608/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019472.049321:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019472.049324:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019472.049327:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334644608:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019472.049345:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334644736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019472.049358:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019472.049364:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019472/real 1734019472] req@ffff8800a78a2680 x1818247334644736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019472.049375:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334644736/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019472.049383:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019472.049385:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019472.049388:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334644736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019472.049393:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334644864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019472.049405:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019472.049411:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019472/real 1734019472] req@ffff8800a78a1880 x1818247334644864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019472.049421:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334644864/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019472.049428:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019472.049430:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019472.049433:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334644864:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019472.284742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019473.286783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019474.288773:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019475.290735:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019476.292799:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019477.056620:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019477.056632:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019477.056638:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019477.056651:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3743 00000100:00080000:2.0:1734019477.056662:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019477.056672:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019477.056723:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334644992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019477.056768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019477.056773:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019477.056775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019477.056780:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3743 00000100:00080000:2.0:1734019477.056784:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019477.056788:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019477.056799:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019477.056812:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334645120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019477.056821:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019477.056824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019477.056826:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019477.056831:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3743 00000100:00080000:2.0:1734019477.056833:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019477.056838:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019477.056842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019477.056852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334645248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019477.056860:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019477.056864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019477.056865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019477.056869:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3743 00000100:00080000:2.0:1734019477.056871:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019477.056875:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019477.056879:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019477.056886:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334644992:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019477.056889:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334645376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019477.056898:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019477.056901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019477.056903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019477.056906:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3743 00000100:00080000:2.0:1734019477.056909:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019477.056912:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019477.056916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019477.056924:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334645504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019477.056932:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019477.056935:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019477.056937:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019477.056941:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3743 00000100:00080000:2.0:1734019477.056944:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019477.056948:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019477.056952:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019477.056960:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334645632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019477.056968:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019477.056972:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734019477.056974:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019477.056974:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019477.056978:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3743 00000100:00080000:2.0:1734019477.056981:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019477.056984:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019477.056988:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019477.056996:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334645760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019477.057000:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019477/real 1734019477] req@ffff8800a78a1880 x1818247334644992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019477.057022:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334644992/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019477.057035:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019477.057039:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019477.057044:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334644992:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019477.057051:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334645120:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019477.057069:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019477.057075:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019477/real 1734019477] req@ffff8800a78a2680 x1818247334645120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019477.057086:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334645120/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019477.057095:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019477.057097:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019477.057101:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334645120:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019477.057107:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334645248:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019477.057121:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019477.057127:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019477/real 1734019477] req@ffff8800a78a1f80 x1818247334645248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019477.057137:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334645248/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019477.057145:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019477.057147:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019477.057151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334645248:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019477.057182:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334645376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019477.057196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019477.057202:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019477/real 1734019477] req@ffff8800a78a3480 x1818247334645376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019477.057213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334645376/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019477.057221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019477.057223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019477.057226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334645376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019477.057232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334645504:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019477.057244:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019477.057250:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019477/real 1734019477] req@ffff8800a78a0700 x1818247334645504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019477.057260:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334645504/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019477.057267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019477.057269:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019477.057272:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334645504:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019477.057278:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334645632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019477.057290:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019477.057297:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019477/real 1734019477] req@ffff8800a78a3b80 x1818247334645632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019477.057308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334645632/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019477.057316:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019477.057318:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019477.057321:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334645632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019477.057326:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334645760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019477.057339:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019477.057345:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019477/real 1734019477] req@ffff8800a78a0a80 x1818247334645760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019477.057355:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334645760/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019477.057362:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019477.057364:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019477.057367:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334645760:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019477.294723:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019478.296690:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019479.298762:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019480.300795:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019481.302688:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019482.064615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019482.064625:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019482.064630:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019482.064641:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3748 00000100:00080000:2.0:1734019482.064650:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019482.064657:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019482.064695:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334645888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019482.064730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019482.064734:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019482.064736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019482.064741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3748 00000100:00080000:2.0:1734019482.064744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019482.064747:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019482.064756:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019482.064768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334646016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019482.064778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019482.064781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019482.064783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019482.064787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3748 00000100:00080000:2.0:1734019482.064790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019482.064793:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019482.064798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019482.064807:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334646144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019482.064815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019482.064819:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019482.064820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019482.064824:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3748 00000100:00080000:2.0:1734019482.064826:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019482.064829:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019482.064833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019482.064842:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334646272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019482.064845:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334645888:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019482.064851:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019482.064855:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019482.064856:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019482.064859:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3748 00000100:00080000:2.0:1734019482.064862:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019482.064866:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019482.064870:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019482.064878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334646400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019482.064885:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019482.064888:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019482.064890:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019482.064893:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3748 00000100:00080000:2.0:1734019482.064896:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019482.064899:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019482.064903:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019482.064910:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334646528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019482.064911:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019482.064918:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019482.064921:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019482.064922:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019482.064925:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3748 00000100:00080000:2.0:1734019482.064928:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019482.064932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019482.064933:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019482/real 1734019482] req@ffff8800a78a0a80 x1818247334645888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019482.064935:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019482.064943:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334646656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019482.064952:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334645888/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019482.064964:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019482.064968:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019482.064972:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334645888:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019482.064980:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334646016:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019482.064996:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019482.065002:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019482/real 1734019482] req@ffff8800a78a3b80 x1818247334646016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019482.065013:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334646016/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019482.065021:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019482.065024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019482.065027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334646016:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019482.065033:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334646144:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019482.065047:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019482.065053:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019482/real 1734019482] req@ffff8800a78a0700 x1818247334646144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019482.065063:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334646144/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019482.065070:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019482.065072:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019482.065076:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334646144:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019482.065101:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334646272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019482.065114:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019482.065120:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019482/real 1734019482] req@ffff8800a78a3480 x1818247334646272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019482.065130:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334646272/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019482.065138:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019482.065140:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019482.065143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334646272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019482.065148:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334646400:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019482.065160:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019482.065166:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019482/real 1734019482] req@ffff8800a78a1f80 x1818247334646400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019482.065176:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334646400/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019482.065183:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019482.065186:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019482.065189:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334646400:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019482.065194:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334646528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019482.065206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019482.065212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019482/real 1734019482] req@ffff8800a78a2680 x1818247334646528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019482.065223:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334646528/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019482.065231:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019482.065233:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019482.065236:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334646528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019482.065241:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334646656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019482.065253:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019482.065259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019482/real 1734019482] req@ffff8800a78a1880 x1818247334646656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019482.065269:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334646656/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019482.065276:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019482.065278:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019482.065281:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334646656:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019482.304742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019483.306770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019484.308744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019485.310775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019486.312729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019487.072612:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019487.072623:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019487.072628:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019487.072639:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3753 00000100:00080000:2.0:1734019487.072647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019487.072655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019487.072693:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334646784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019487.072727:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019487.072731:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019487.072733:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019487.072738:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3753 00000100:00080000:2.0:1734019487.072741:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019487.072745:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019487.072755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019487.072766:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334646912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019487.072776:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019487.072779:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019487.072781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019487.072785:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3753 00000100:00080000:2.0:1734019487.072787:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019487.072791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019487.072796:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019487.072805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334647040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019487.072814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019487.072817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019487.072818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019487.072822:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3753 00000100:00080000:2.0:1734019487.072824:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019487.072827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019487.072831:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019487.072835:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334646784:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019487.072839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334647168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019487.072848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019487.072851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019487.072852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019487.072856:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3753 00000100:00080000:2.0:1734019487.072858:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019487.072861:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019487.072865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019487.072873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334647296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019487.072881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019487.072884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019487.072886:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019487.072890:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3753 00000100:00080000:2.0:1734019487.072893:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019487.072896:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019487.072900:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019487.072907:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334647424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019487.072915:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019487.072918:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019487.072919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019487.072922:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3753 00000100:00080000:2.0:1734019487.072925:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019487.072926:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334646912:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019487.072928:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019487.072932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019487.072939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334647552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019487.072947:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334647040:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019487.072969:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334647168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019487.072985:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334647296:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019487.073000:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334647424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019487.073014:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334647552:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019487.314704:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019488.316779:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019489.318630:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019490.078624:0:1968:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019490.078652:0:1968:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019490.078666:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019490.078688:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019490.078693:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019490.078697:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019490.078701:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019490.078705:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019490.078708:0:1968:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734019490.078755:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019487/real 1734019490] req@ffff8800a78a3480 x1818247334647168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019490.078773:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334647168/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019490.078780:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019490.078783:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019490.078786:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334647168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019490.078790:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019487/real 1734019490] req@ffff8800a78a0700 x1818247334647296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019490.078796:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334647296/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019490.078800:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019490.078802:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019490.078804:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334647296:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019490.078807:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019487/real 1734019490] req@ffff8800a78a3b80 x1818247334647424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019490.078818:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334647424/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019490.078823:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019490.078824:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019490.078826:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334647424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019490.078829:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019487/real 1734019490] req@ffff8800a78a0a80 x1818247334647552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019490.078835:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334647552/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019490.078838:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019490.078840:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019490.078841:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334647552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019490.078844:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019487/real 1734019490] req@ffff8800a78a1880 x1818247334646784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019490.078850:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334646784/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019490.078854:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019490.078855:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019490.078857:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334646784:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019490.078860:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019487/real 1734019490] req@ffff8800a78a2680 x1818247334646912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019490.078866:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334646912/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019490.078873:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019490.078875:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019490.078876:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334646912:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019490.078880:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019487/real 1734019490] req@ffff8800a78a1f80 x1818247334647040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019490.078889:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334647040/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019490.078895:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019490.078897:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019490.078900:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334647040:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019490.320612:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019491.322706:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019492.080579:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019492.080590:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019492.080594:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019492.080605:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3758 00000100:00080000:2.0:1734019492.080616:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019492.080623:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019492.080662:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334647680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019492.080705:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019492.080709:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019492.080711:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019492.080715:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3758 00000100:00080000:2.0:1734019492.080718:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019492.080722:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019492.080731:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019492.080742:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334647808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019492.080751:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019492.080754:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019492.080756:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019492.080760:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3758 00000100:00080000:2.0:1734019492.080762:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019492.080766:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019492.080771:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1734019492.080774:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334647680:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019492.080780:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334647936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019492.080789:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019492.080792:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019492.080794:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019492.080797:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3758 00000100:00080000:2.0:1734019492.080799:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019492.080803:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019492.080807:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019492.080815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334648064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019492.080823:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019492.080826:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019492.080827:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019492.080831:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3758 00000100:00080000:2.0:1734019492.080835:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019492.080838:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019492.080842:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734019492.080848:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019492.080849:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334648192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019492.080857:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019492.080860:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019492.080861:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019492.080864:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3758 00000100:00080000:2.0:1734019492.080869:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019492.080872:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019492.080876:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019492.080877:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019492/real 1734019492] req@ffff8800a78a1f80 x1818247334647680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019492.080884:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334648320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019492.080892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019492.080896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019492.080897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019492.080901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3758 00000100:00080000:2.0:1734019492.080904:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019492.080906:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334647680/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019492.080907:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019492.080911:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019492.080918:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334648448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019492.080921:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019492.080925:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019492.080933:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334647680:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019492.080955:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334647808:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019492.080984:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019492.080997:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019492/real 1734019492] req@ffff8800a78a2680 x1818247334647808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019492.081014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334647808/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019492.081027:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019492.081031:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019492.081037:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334647808:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019492.081067:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334647936:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019492.081094:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019492.081106:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019492/real 1734019492] req@ffff8800a78a1880 x1818247334647936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019492.081125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334647936/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019492.081134:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019492.081138:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019492.081143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334647936:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019492.081153:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334648064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019492.081229:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019492.081240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019492/real 1734019492] req@ffff8800a78a0a80 x1818247334648064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019492.081259:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334648064/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019492.081267:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019492.081270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019492.081273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334648064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019492.081280:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334648192:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019492.081297:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019492.081304:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019492/real 1734019492] req@ffff8800a78a3b80 x1818247334648192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019492.081315:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334648192/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019492.081323:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019492.081326:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019492.081331:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334648192:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019492.081340:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334648320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019492.081363:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019492.081380:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019492/real 1734019492] req@ffff8800a78a0700 x1818247334648320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019492.081398:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334648320/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019492.081407:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019492.081410:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019492.081413:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334648320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019492.081420:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334648448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019492.081436:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019492.081442:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019492/real 1734019492] req@ffff8800a78a3480 x1818247334648448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019492.081452:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334648448/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019492.081460:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019492.081792:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019492.081800:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334648448:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019492.324835:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019493.326736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019494.328778:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019495.330715:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019496.332698:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019497.088598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019497.088609:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019497.088614:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019497.088624:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3763 00000100:00080000:2.0:1734019497.088633:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019497.088640:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019497.088678:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334648576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019497.088712:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019497.088716:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019497.088719:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019497.088723:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3763 00000100:00080000:2.0:1734019497.088726:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019497.088730:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019497.088739:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019497.088751:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334648704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019497.088762:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019497.088765:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019497.088766:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019497.088770:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3763 00000100:00080000:2.0:1734019497.088772:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019497.088776:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019497.088780:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019497.088789:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334648832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019497.088797:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019497.088800:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019497.088802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019497.088805:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3763 00000100:00080000:2.0:1734019497.088807:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019497.088810:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019497.088814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019497.088819:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334648576:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019497.088822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334648960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019497.088832:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019497.088835:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019497.088836:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019497.088840:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3763 00000100:00080000:2.0:1734019497.088843:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019497.088846:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019497.088850:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019497.088858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334649088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019497.088866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019497.088868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019497.088870:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019497.088873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3763 00000100:00080000:2.0:1734019497.088876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019497.088879:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019497.088883:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019497.088883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019497.088891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334649216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019497.088899:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019497.088902:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019497.088903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019497.088904:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019497/real 1734019497] req@ffff8800a78a3480 x1818247334648576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019497.088907:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3763 00000100:00080000:2.0:1734019497.088909:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019497.088912:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019497.088916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019497.088923:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334648576/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019497.088924:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334649344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019497.088934:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019497.088937:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019497.088942:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334648576:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019497.088950:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334648704:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019497.088967:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019497.088974:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019497/real 1734019497] req@ffff8800a78a0700 x1818247334648704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019497.088986:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334648704/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019497.088994:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019497.088997:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019497.089000:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334648704:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019497.089006:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334648832:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019497.089019:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019497.089025:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019497/real 1734019497] req@ffff8800a78a3b80 x1818247334648832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019497.089035:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334648832/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019497.089042:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019497.089045:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019497.089048:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334648832:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019497.089071:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334648960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019497.089085:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019497.089124:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019497/real 1734019497] req@ffff8800a78a0a80 x1818247334648960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019497.089136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334648960/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019497.089144:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019497.089146:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019497.089149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334648960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019497.089155:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334649088:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019497.089170:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019497.089176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019497/real 1734019497] req@ffff8800a78a1880 x1818247334649088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019497.089187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334649088/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019497.089195:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019497.089197:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019497.089200:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334649088:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019497.089206:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334649216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019497.089217:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019497.089223:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019497/real 1734019497] req@ffff8800a78a2680 x1818247334649216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019497.089233:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334649216/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019497.089240:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019497.089243:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019497.089246:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334649216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019497.089252:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334649344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019497.089264:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019497.089270:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019497/real 1734019497] req@ffff8800a78a1f80 x1818247334649344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019497.089280:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334649344/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019497.089287:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019497.089289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019497.089292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334649344:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019497.334758:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019498.336750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019499.338719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019500.340811:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019501.342736:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019502.096610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019502.096621:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019502.096625:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019502.096635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3768 00000100:00080000:2.0:1734019502.096644:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019502.096652:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019502.096688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334649472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019502.096723:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019502.096727:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019502.096729:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019502.096734:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3768 00000100:00080000:2.0:1734019502.096737:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019502.096741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019502.096751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019502.096762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334649600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019502.096772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019502.096775:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019502.096777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019502.096780:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3768 00000100:00080000:2.0:1734019502.096783:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019502.096787:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019502.096792:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019502.096801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334649728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019502.096810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019502.096813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019502.096814:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019502.096817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3768 00000100:00080000:2.0:1734019502.096820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019502.096823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019502.096827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019502.096836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334649856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019502.096838:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334649472:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019502.096844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019502.096847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019502.096849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019502.096852:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3768 00000100:00080000:2.0:1734019502.096855:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019502.096858:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019502.096862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019502.096870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334649984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019502.096877:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019502.096880:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019502.096882:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019502.096886:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3768 00000100:00080000:2.0:1734019502.096898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019502.096902:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019502.096903:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019502.096906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019502.096914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334650112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019502.096922:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019502.096924:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019502/real 1734019502] req@ffff8800a78a1f80 x1818247334649472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019502.096925:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019502.096927:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019502.096930:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3768 00000100:00080000:2.0:1734019502.096933:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019502.096936:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019502.096940:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019502.096945:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334649472/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019502.096947:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334650240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019502.096955:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019502.096959:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019502.096964:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334649472:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019502.096974:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334649600:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019502.096992:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019502.096999:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019502/real 1734019502] req@ffff8800a78a2680 x1818247334649600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019502.097062:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334649600/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019502.097071:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019502.097074:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019502.097078:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334649600:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019502.097085:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334649728:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019502.097103:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019502.097110:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019502/real 1734019502] req@ffff8800a78a1880 x1818247334649728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019502.097121:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334649728/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019502.097129:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019502.097132:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019502.097135:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334649728:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019502.097160:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334649856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019502.097173:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019502.097180:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019502/real 1734019502] req@ffff8800a78a0a80 x1818247334649856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019502.097191:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334649856/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019502.097200:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019502.097202:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019502.097205:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334649856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019502.097210:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334649984:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019502.097222:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019502.097228:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019502/real 1734019502] req@ffff8800a78a3b80 x1818247334649984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019502.097238:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334649984/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019502.097245:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019502.097247:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019502.097250:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334649984:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019502.097256:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334650112:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019502.097268:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019502.097274:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019502/real 1734019502] req@ffff8800a78a0700 x1818247334650112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019502.097284:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334650112/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019502.097291:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019502.097294:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019502.097297:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334650112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019502.097303:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334650240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019502.097315:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019502.097321:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019502/real 1734019502] req@ffff8800a78a3480 x1818247334650240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019502.097332:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334650240/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019502.097339:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019502.097342:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019502.097345:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334650240:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019502.344712:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019503.346703:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019504.348775:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019505.350719:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019506.352704:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019507.104569:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019507.104579:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019507.104584:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019507.104595:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3773 00000100:00080000:2.0:1734019507.104603:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019507.104611:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019507.104647:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334650368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019507.104691:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019507.104697:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019507.104700:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019507.104706:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3773 00000100:00080000:2.0:1734019507.104711:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019507.104717:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019507.104729:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019507.104747:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334650496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019507.104760:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334650368:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019507.104763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019507.104767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019507.104770:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019507.104775:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3773 00000100:00080000:2.0:1734019507.104779:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019507.104785:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019507.104791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019507.104804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334650624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019507.104816:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019507.104820:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019507.104823:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019507.104828:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3773 00000800:00000100:1.0:1734019507.104829:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019507.104832:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019507.104837:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019507.104843:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019507.104853:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019507/real 1734019507] req@ffff8800a78a3480 x1818247334650368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019507.104855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334650752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019507.104869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019507.104873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734019507.104876:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334650368/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019507.104876:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019507.104881:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3773 00000100:00080000:2.0:1734019507.104885:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019507.104890:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734019507.104891:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019507.104895:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000008:00100000:2.0:1734019507.104896:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019507.104902:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334650368:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019507.104908:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334650880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019507.104970:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019507.104976:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019507.104978:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019507.104981:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334650496:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019507.104985:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3773 00000100:00080000:2.0:1734019507.104989:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019507.104995:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019507.105002:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019507.105013:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019507.105017:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334651008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019507.105026:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019507/real 1734019507] req@ffff8800a78a0700 x1818247334650496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019507.105031:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019507.105037:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019507.105040:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019507.105045:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334650496/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019507.105045:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3773 00000100:00080000:2.0:1734019507.105049:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019507.105054:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734019507.105058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1734019507.105060:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734019507.105062:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019507.105068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334650496:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019507.105072:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334651136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019507.105078:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334650624:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019507.105102:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019507.105114:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019507/real 1734019507] req@ffff8800a78a3b80 x1818247334650624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019507.105131:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334650624/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019507.105144:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019507.105148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019507.105153:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334650624:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019507.105163:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334650752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019507.105186:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019507.105197:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019507/real 1734019507] req@ffff8800a78a0a80 x1818247334650752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019507.105216:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334650752/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019507.105226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019507.105228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019507.105232:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334650752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019507.105238:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334650880:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019507.105255:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019507.105262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019507/real 1734019507] req@ffff8800a78a1880 x1818247334650880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019507.105272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334650880/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019507.105280:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019507.105282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019507.105286:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334650880:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019507.105304:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334651008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019507.105316:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019507.105322:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019507/real 1734019507] req@ffff8800a78a2680 x1818247334651008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019507.105333:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334651008/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019507.105340:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019507.105344:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019507.105347:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334651008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019507.105353:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334651136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019507.105366:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019507.105372:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019507/real 1734019507] req@ffff8800a78a1f80 x1818247334651136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019507.105382:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334651136/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019507.105388:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019507.105391:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019507.105393:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334651136:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019507.354722:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019508.356617:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019509.358708:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019510.360698:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019511.362698:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019512.112624:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019512.112638:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019512.112644:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019512.112658:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3778 00000100:00080000:2.0:1734019512.112669:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019512.112679:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019512.112732:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334651264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019512.112773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019512.112777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019512.112779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019512.112784:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3778 00000100:00080000:2.0:1734019512.112787:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019512.112791:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019512.112803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019512.112815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334651392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019512.112826:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019512.112830:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019512.112832:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019512.112867:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3778 00000100:00080000:2.0:1734019512.112870:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019512.112874:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019512.112879:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019512.112891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334651520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019512.112901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019512.112905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019512.112907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019512.112910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3778 00000100:00080000:2.0:1734019512.112913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019512.112916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019512.112920:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019512.112929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334651648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019512.112935:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334651264:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019512.112939:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019512.112943:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019512.112944:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019512.112948:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3778 00000100:00080000:2.0:1734019512.112952:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019512.112955:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019512.112960:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019512.112969:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334651776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019512.112977:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019512.112980:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019512.112981:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019512.112985:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3778 00000100:00080000:2.0:1734019512.112988:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019512.112992:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019512.112996:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019512.113004:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334651904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019512.113006:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019512.113011:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019512.113015:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019512.113016:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019512.113020:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3778 00000100:00080000:2.0:1734019512.113024:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019512.113027:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019512.113029:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019512/real 1734019512] req@ffff8800a78a1f80 x1818247334651264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019512.113032:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019512.113040:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334652032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019512.113050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334651264/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019512.113061:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019512.113065:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019512.113070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334651264:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019512.113082:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334651392:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019512.113101:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019512.113108:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019512/real 1734019512] req@ffff8800a78a2680 x1818247334651392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019512.113121:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334651392/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019512.113129:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019512.113132:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019512.113136:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334651392:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019512.113142:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334651520:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019512.113156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019512.113162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019512/real 1734019512] req@ffff8800a78a1880 x1818247334651520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019512.113172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334651520/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019512.113180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019512.113182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019512.113185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334651520:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019512.113213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334651648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019512.113227:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019512.113234:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019512/real 1734019512] req@ffff8800a78a0a80 x1818247334651648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019512.113245:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334651648/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019512.113254:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019512.113256:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019512.113259:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334651648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019512.113265:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334651776:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019512.113278:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019512.113284:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019512/real 1734019512] req@ffff8800a78a3b80 x1818247334651776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019512.113294:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334651776/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019512.113301:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019512.113303:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019512.113306:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334651776:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019512.113313:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334651904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019512.113325:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019512.113331:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019512/real 1734019512] req@ffff8800a78a0700 x1818247334651904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019512.113342:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334651904/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019512.113348:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019512.113352:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019512.113355:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334651904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019512.113362:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334652032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019512.113375:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019512.113381:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019512/real 1734019512] req@ffff8800a78a3480 x1818247334652032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019512.113391:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334652032/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019512.113399:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019512.113401:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019512.113405:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334652032:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019512.364702:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019513.366744:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019514.368740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019515.370740:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019516.372727:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019517.120617:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019517.120627:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019517.120632:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019517.120642:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3783 00000100:00080000:2.0:1734019517.120651:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019517.120658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019517.120696:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334652160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019517.120730:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019517.120735:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019517.120737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019517.120741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3783 00000100:00080000:2.0:1734019517.120744:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019517.120748:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019517.120802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019517.120818:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334652288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019517.120830:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019517.120833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1734019517.120834:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334652160:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019517.120835:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019517.120839:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3783 00000100:00080000:2.0:1734019517.120843:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019517.120846:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019517.120851:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019517.120860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334652416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019517.120869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019517.120873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019517.120874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019517.120877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3783 00000100:00080000:2.0:1734019517.120880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019517.120883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019517.120887:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019517.120895:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334652544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019517.120898:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019517.120903:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019517.120906:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019517.120908:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019517.120911:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3783 00000100:00080000:2.0:1734019517.120915:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019517.120918:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019517.120919:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019517/real 1734019517] req@ffff8800a78a3480 x1818247334652160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019517.120922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019517.120929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334652672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019517.120937:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019517.120938:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334652160/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019517.120940:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019517.120942:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019517.120946:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3783 00000100:00080000:1.0:1734019517.120949:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019517.120949:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734019517.120952:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019517.120952:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019517.120956:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019517.120957:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334652160:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019517.120964:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334652800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019517.120971:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019517.120974:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019517.120976:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019517.120979:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3783 00000100:00100000:1.0:1734019517.120980:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334652288:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019517.120982:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019517.120985:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019517.120988:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019517.120996:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334652928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019517.120998:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019517.121005:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019517/real 1734019517] req@ffff8800a78a0700 x1818247334652288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019517.121017:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334652288/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019517.121026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019517.121028:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019517.121032:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334652288:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019517.121038:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334652416:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019517.121051:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019517.121058:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019517/real 1734019517] req@ffff8800a78a3b80 x1818247334652416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019517.121070:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334652416/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019517.121078:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019517.121080:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019517.121083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334652416:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019517.121088:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334652544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019517.121101:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019517.121107:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019517/real 1734019517] req@ffff8800a78a0a80 x1818247334652544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019517.121118:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334652544/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019517.121126:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019517.121128:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019517.121131:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334652544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019517.121136:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334652672:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019517.121149:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019517.121155:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019517/real 1734019517] req@ffff8800a78a1880 x1818247334652672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019517.121165:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334652672/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019517.121172:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019517.121174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019517.121177:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334652672:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019517.121182:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334652800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019517.121194:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019517.121200:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019517/real 1734019517] req@ffff8800a78a2680 x1818247334652800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019517.121209:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334652800/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019517.121216:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019517.121219:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019517.121222:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334652800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019517.121240:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334652928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019517.121253:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019517.121259:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019517/real 1734019517] req@ffff8800a78a1f80 x1818247334652928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019517.121270:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334652928/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019517.121277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019517.121279:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019517.121282:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334652928:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019517.374734:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019518.376826:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019519.378796:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019520.380770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019521.382763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019522.128626:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019522.128636:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019522.128641:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019522.128651:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3788 00000100:00080000:2.0:1734019522.128660:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019522.128705:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019522.128745:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334653056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019522.128786:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019522.128790:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019522.128792:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019522.128796:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3788 00000100:00080000:2.0:1734019522.128800:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019522.128804:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019522.128814:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019522.128826:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334653184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019522.128835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019522.128838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019522.128840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019522.128843:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3788 00000100:00080000:2.0:1734019522.128846:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019522.128849:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019522.128854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019522.128862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334653312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019522.128871:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019522.128874:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019522.128875:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019522.128879:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3788 00000100:00080000:2.0:1734019522.128881:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019522.128885:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019522.128889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019522.128897:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334653440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019522.128902:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334653056:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019522.128906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019522.128909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019522.128911:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019522.128914:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3788 00000100:00080000:2.0:1734019522.128916:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019522.128919:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019522.128924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019522.128931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334653568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019522.128939:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019522.128942:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019522.128944:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019522.128947:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3788 00000100:00080000:2.0:1734019522.128950:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019522.128953:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019522.128957:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019522.128965:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334653696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019522.128967:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019522.128973:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019522.128977:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019522.128978:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019522.128982:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3788 00000100:00080000:2.0:1734019522.128985:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019522.128988:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019522.128989:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019522/real 1734019522] req@ffff8800a78a1f80 x1818247334653056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019522.128992:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019522.129000:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334653824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019522.129008:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334653056/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019522.129018:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019522.129021:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019522.129026:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334653056:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019522.129037:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334653184:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019522.129053:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019522.129060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019522/real 1734019522] req@ffff8800a78a2680 x1818247334653184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019522.129072:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334653184/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019522.129080:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019522.129082:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019522.129086:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334653184:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019522.129091:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334653312:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019522.129104:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019522.129111:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019522/real 1734019522] req@ffff8800a78a1880 x1818247334653312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019522.129121:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334653312/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019522.129128:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019522.129130:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019522.129133:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334653312:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019522.129157:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334653440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019522.129169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019522.129176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019522/real 1734019522] req@ffff8800a78a0a80 x1818247334653440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019522.129187:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334653440/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019522.129194:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019522.129197:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019522.129200:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334653440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019522.129205:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334653568:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019522.129217:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019522.129223:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019522/real 1734019522] req@ffff8800a78a3b80 x1818247334653568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019522.129233:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334653568/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019522.129240:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019522.129242:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019522.129245:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334653568:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019522.129250:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334653696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019522.129262:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019522.129268:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019522/real 1734019522] req@ffff8800a78a0700 x1818247334653696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019522.129278:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334653696/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019522.129285:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019522.129288:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019522.129291:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334653696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019522.129297:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334653824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019522.129310:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019522.129316:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019522/real 1734019522] req@ffff8800a78a3480 x1818247334653824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019522.129326:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334653824/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019522.129333:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019522.129335:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019522.129338:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334653824:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019522.384737:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019523.386775:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019524.388744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019525.390798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019526.392759:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019527.136631:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019527.136645:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019527.136651:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019527.136665:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3793 00000100:00080000:2.0:1734019527.136676:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019527.136686:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019527.136737:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334653952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019527.136775:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019527.136779:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019527.136782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019527.136786:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3793 00000100:00080000:2.0:1734019527.136790:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019527.136794:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019527.136806:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019527.136818:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334654080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019527.136829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019527.136833:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019527.136834:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019527.136838:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3793 00000100:00080000:2.0:1734019527.136841:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019527.136844:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019527.136849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019527.136858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334654208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019527.136869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019527.136872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019527.136873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019527.136877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3793 00000100:00080000:2.0:1734019527.136880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019527.136883:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019527.136888:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019527.136897:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334654336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019527.136906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019527.136909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019527.136910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019527.136914:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3793 00000100:00080000:2.0:1734019527.136918:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019527.136919:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334653952:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019527.136922:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019527.136926:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019527.136935:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334654464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019527.136943:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019527.136946:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019527.136947:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019527.136951:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3793 00000100:00080000:2.0:1734019527.136954:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019527.136957:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019527.136962:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019527.136970:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334654592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019527.136978:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019527.136981:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019527.136983:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019527.136986:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3793 00000100:00080000:2.0:1734019527.136990:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019527.136993:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019527.136994:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019527.136998:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019527.137006:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334654720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019527.137018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019527/real 1734019527] req@ffff8800a78a3480 x1818247334653952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019527.137038:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334653952/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019527.137049:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019527.137052:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019527.137057:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334653952:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019527.137066:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334654080:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019527.137084:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019527.137091:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019527/real 1734019527] req@ffff8800a78a0700 x1818247334654080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019527.137103:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334654080/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019527.137111:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019527.137114:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019527.137117:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334654080:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019527.137124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334654208:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019527.137138:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019527.137144:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019527/real 1734019527] req@ffff8800a78a3b80 x1818247334654208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019527.137154:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334654208/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019527.137162:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019527.137164:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019527.137167:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334654208:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019527.137192:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334654336:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019527.137206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019527.137212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019527/real 1734019527] req@ffff8800a78a0a80 x1818247334654336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019527.137223:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334654336/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019527.137231:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019527.137233:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019527.137236:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334654336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019527.137243:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334654464:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019527.137256:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019527.137262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019527/real 1734019527] req@ffff8800a78a1880 x1818247334654464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019527.137272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334654464/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019527.137279:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019527.137282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019527.137285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334654464:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019527.137291:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334654592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019527.137303:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019527.137309:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019527/real 1734019527] req@ffff8800a78a2680 x1818247334654592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019527.137319:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334654592/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019527.137326:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019527.137329:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019527.137332:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334654592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019527.137339:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334654720:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019527.137351:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019527.137357:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019527/real 1734019527] req@ffff8800a78a1f80 x1818247334654720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019527.137367:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334654720/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019527.137374:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019527.137376:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019527.137379:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334654720:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019527.394798:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019528.396757:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019529.398783:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019530.400785:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019531.402738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019532.144601:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019532.144614:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019532.144621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019532.144635:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3798 00000100:00080000:2.0:1734019532.144645:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019532.144655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019532.144707:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334654848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019532.144747:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019532.144751:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019532.144753:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019532.144759:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3798 00000100:00080000:2.0:1734019532.144762:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019532.144766:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019532.144777:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019532.144790:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334654976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019532.144800:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019532.144803:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019532.144805:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019532.144809:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3798 00000100:00080000:2.0:1734019532.144812:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019532.144816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019532.144822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019532.144832:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334655104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019532.144842:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019532.144845:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019532.144846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019532.144850:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3798 00000100:00080000:2.0:1734019532.144853:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019532.144856:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019532.144860:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019532.144869:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334655232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019532.144878:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019532.144881:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734019532.144883:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334654848:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019532.144883:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019532.144886:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3798 00000100:00080000:2.0:1734019532.144889:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019532.144892:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019532.144897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019532.144906:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334655360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019532.144914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019532.144917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019532.144919:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019532.144924:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3798 00000100:00080000:2.0:1734019532.144926:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019532.144930:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019532.144934:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019532.144942:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334655488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019532.144950:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734019532.144953:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019532.144953:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019532.144955:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019532.144959:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3798 00000100:00080000:2.0:1734019532.144962:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019532.144965:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019532.144969:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019532.144975:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019532/real 1734019532] req@ffff8800a78a1f80 x1818247334654848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019532.144978:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334655616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019532.144996:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334654848/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019532.145008:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019532.145012:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019532.145016:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334654848:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019532.145028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334654976:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019532.145046:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019532.145053:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019532/real 1734019532] req@ffff8800a78a2680 x1818247334654976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019532.145065:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334654976/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019532.145073:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019532.145076:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019532.145079:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334654976:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019532.145086:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334655104:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019532.145099:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019532.145106:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019532/real 1734019532] req@ffff8800a78a1880 x1818247334655104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019532.145116:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334655104/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019532.145123:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019532.145126:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019532.145129:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334655104:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019532.145155:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334655232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019532.145168:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019532.145174:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019532/real 1734019532] req@ffff8800a78a0a80 x1818247334655232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019532.145185:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334655232/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019532.145193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019532.145195:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019532.145198:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334655232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019532.145204:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334655360:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019532.145216:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019532.145222:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019532/real 1734019532] req@ffff8800a78a3b80 x1818247334655360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019532.145232:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334655360/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019532.145239:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019532.145242:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019532.145245:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334655360:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019532.145251:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334655488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019532.145263:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019532.145269:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019532/real 1734019532] req@ffff8800a78a0700 x1818247334655488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019532.145280:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334655488/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019532.145286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019532.145289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019532.145292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334655488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019532.145299:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334655616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019532.145312:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019532.145318:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019532/real 1734019532] req@ffff8800a78a3480 x1818247334655616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019532.145328:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334655616/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019532.145335:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019532.145337:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019532.145340:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334655616:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019532.404719:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019533.406776:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019534.408744:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019535.410757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019536.412740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019537.152615:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019537.152625:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019537.152631:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019537.152644:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3803 00000100:00080000:2.0:1734019537.152654:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019537.152664:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019537.152712:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334655744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019537.152757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019537.152762:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019537.152764:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019537.152769:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3803 00000100:00080000:2.0:1734019537.152772:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019537.152777:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019537.152788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019537.152800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334655872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019537.152810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019537.152813:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019537.152815:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019537.152819:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3803 00000100:00080000:2.0:1734019537.152822:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019537.152826:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019537.152830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019537.152841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334656000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019537.152849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019537.152853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019537.152854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019537.152858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3803 00000100:00080000:2.0:1734019537.152860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019537.152865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019537.152869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019537.152878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334656128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019537.152881:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334655744:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019537.152888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019537.152891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019537.152892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019537.152896:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3803 00000100:00080000:2.0:1734019537.152899:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019537.152902:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019537.152906:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019537.152915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334656256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019537.152923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019537.152926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019537.152927:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019537.152931:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3803 00000100:00080000:2.0:1734019537.152934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019537.152938:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019537.152942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019537.152950:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019537.152950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334656384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019537.152959:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019537.152963:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019537.152965:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019537.152968:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3803 00000100:00080000:2.0:1734019537.152972:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019537.152973:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019537/real 1734019537] req@ffff8800a78a3480 x1818247334655744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019537.152976:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019537.152980:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019537.152988:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334656512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019537.152993:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334655744/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019537.153004:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019537.153008:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019537.153013:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334655744:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019537.153022:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334655872:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019537.153040:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019537.153047:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019537/real 1734019537] req@ffff8800a78a0700 x1818247334655872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019537.153058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334655872/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019537.153066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019537.153069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019537.153072:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334655872:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019537.153079:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334656000:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019537.153092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019537.153098:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019537/real 1734019537] req@ffff8800a78a3b80 x1818247334656000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019537.153109:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334656000/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019537.153117:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019537.153119:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019537.153122:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334656000:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019537.153149:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334656128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019537.153163:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019537.153169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019537/real 1734019537] req@ffff8800a78a0a80 x1818247334656128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019537.153180:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334656128/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019537.153188:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019537.153191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019537.153194:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334656128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019537.153200:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334656256:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019537.153212:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019537.153218:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019537/real 1734019537] req@ffff8800a78a1880 x1818247334656256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019537.153228:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334656256/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019537.153236:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019537.153238:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019537.153241:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334656256:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019537.153248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334656384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019537.153260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019537.153266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019537/real 1734019537] req@ffff8800a78a2680 x1818247334656384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019537.153276:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334656384/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019537.153283:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019537.153286:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019537.153289:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334656384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019537.153295:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334656512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019537.153308:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019537.153314:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019537/real 1734019537] req@ffff8800a78a1f80 x1818247334656512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019537.153324:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334656512/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019537.153331:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019537.153333:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019537.153336:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334656512:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019537.414739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019538.416727:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019539.418754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019540.420735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019541.422732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019542.160582:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019542.160592:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019542.160597:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019542.160608:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3808 00000100:00080000:2.0:1734019542.160616:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019542.160624:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019542.160662:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334656640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019542.160694:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019542.160698:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019542.160700:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019542.160704:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3808 00000100:00080000:2.0:1734019542.160707:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019542.160711:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019542.160720:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019542.160731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334656768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019542.160741:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019542.160745:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019542.160747:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019542.160750:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3808 00000100:00080000:2.0:1734019542.160753:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019542.160756:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019542.160761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019542.160769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334656896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019542.160778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019542.160781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019542.160782:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019542.160786:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3808 00000100:00080000:2.0:1734019542.160788:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019542.160791:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019542.160795:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019542.160804:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334657024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019542.160805:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334656640:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019542.160813:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019542.160816:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019542.160817:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019542.160820:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3808 00000100:00080000:2.0:1734019542.160823:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019542.160827:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019542.160830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019542.160838:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334657152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019542.160846:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019542.160849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019542.160850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019542.160853:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3808 00000100:00080000:2.0:1734019542.160865:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019542.160868:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019542.160869:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019542.160872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019542.160880:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334657280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019542.160888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019542.160889:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019542/real 1734019542] req@ffff8800a78a1f80 x1818247334656640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019542.160891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019542.160893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019542.160896:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3808 00000100:00080000:2.0:1734019542.160900:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019542.160903:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019542.160907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019542.160908:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334656640/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019542.160914:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334657408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019542.160918:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019542.160921:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019542.160926:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334656640:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019542.160937:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334656768:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019542.160954:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019542.160961:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019542/real 1734019542] req@ffff8800a78a2680 x1818247334656768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019542.160973:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334656768/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019542.160980:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019542.160983:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019542.160986:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334656768:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019542.160992:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334656896:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019542.161006:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019542.161012:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019542/real 1734019542] req@ffff8800a78a1880 x1818247334656896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019542.161022:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334656896/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019542.161029:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019542.161031:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019542.161034:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334656896:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019542.161058:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334657024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019542.161071:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019542.161077:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019542/real 1734019542] req@ffff8800a78a0a80 x1818247334657024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019542.161088:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334657024/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019542.161096:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019542.161098:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019542.161101:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334657024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019542.161106:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334657152:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019542.161118:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019542.161124:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019542/real 1734019542] req@ffff8800a78a3b80 x1818247334657152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019542.161134:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334657152/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019542.161141:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019542.161143:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019542.161146:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334657152:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019542.161152:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334657280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019542.161164:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019542.161170:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019542/real 1734019542] req@ffff8800a78a0700 x1818247334657280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019542.161180:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334657280/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019542.161187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019542.161190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019542.161193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334657280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019542.161199:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334657408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019542.161211:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019542.161217:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019542/real 1734019542] req@ffff8800a78a3480 x1818247334657408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019542.161227:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334657408/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019542.161235:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019542.161237:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019542.161240:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334657408:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019542.424746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019543.426743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019544.428625:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019545.430701:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019546.432761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019547.168572:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019547.168582:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019547.168587:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019547.168596:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3813 00000100:00080000:2.0:1734019547.168605:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019547.168613:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019547.168651:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334657536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019547.168686:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019547.168690:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019547.168692:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019547.168697:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3813 00000100:00080000:2.0:1734019547.168700:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019547.168704:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019547.168713:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019547.168725:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334657664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019547.168734:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019547.168737:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019547.168739:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019547.168750:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3813 00000100:00080000:2.0:1734019547.168753:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019547.168756:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019547.168762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019547.168771:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334657792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019547.168780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019547.168783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019547.168785:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019547.168788:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3813 00000100:00080000:2.0:1734019547.168791:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019547.168794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019547.168797:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019547.168805:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334657920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019547.168812:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334657536:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019547.168814:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019547.168817:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019547.168818:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019547.168822:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3813 00000100:00080000:2.0:1734019547.168824:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019547.168827:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019547.168831:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019547.168838:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334658048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019547.168846:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019547.168849:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019547.168850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019547.168854:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3813 00000100:00080000:2.0:1734019547.168857:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019547.168860:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019547.168864:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019547.168871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334658176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019547.168875:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019547.168879:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019547.168882:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019547.168884:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019547.168887:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3813 00000100:00080000:2.0:1734019547.168891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019547.168894:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019547.168896:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019547/real 1734019547] req@ffff8800a78a3480 x1818247334657536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019547.168898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019547.168905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334658304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019547.168916:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334657536/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019547.168926:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019547.168929:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019547.168934:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334657536:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019547.168942:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334657664:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019547.168960:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019547.168967:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019547/real 1734019547] req@ffff8800a78a0700 x1818247334657664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019547.168978:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334657664/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019547.168986:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019547.168989:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019547.168992:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334657664:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019547.168997:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334657792:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019547.169010:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019547.169016:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019547/real 1734019547] req@ffff8800a78a3b80 x1818247334657792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019547.169026:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334657792/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019547.169034:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019547.169036:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019547.169039:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334657792:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019547.169063:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334657920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019547.169076:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019547.169082:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019547/real 1734019547] req@ffff8800a78a0a80 x1818247334657920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019547.169093:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334657920/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019547.169100:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019547.169103:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019547.169106:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334657920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019547.169111:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334658048:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019547.169123:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019547.169129:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019547/real 1734019547] req@ffff8800a78a1880 x1818247334658048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019547.169139:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334658048/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019547.169146:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019547.169148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019547.169151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334658048:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019547.169157:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334658176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019547.169169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019547.169175:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019547/real 1734019547] req@ffff8800a78a2680 x1818247334658176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019547.169184:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334658176/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019547.169191:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019547.169194:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019547.169197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334658176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019547.169203:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334658304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019547.169216:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019547.169221:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019547/real 1734019547] req@ffff8800a78a1f80 x1818247334658304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019547.169231:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334658304/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019547.169238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019547.169240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019547.169243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334658304:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019547.434745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019548.436709:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019549.438743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019550.440713:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019551.442748:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019552.176620:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019552.176633:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019552.176639:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019552.176653:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3818 00000100:00080000:2.0:1734019552.176664:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019552.176673:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019552.176724:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334658432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019552.176772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019552.176777:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019552.176779:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019552.176784:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3818 00000100:00080000:2.0:1734019552.176788:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019552.176792:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019552.176803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019552.176816:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334658560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019552.176825:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019552.176828:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019552.176830:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019552.176834:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3818 00000100:00080000:2.0:1734019552.176837:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019552.176841:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019552.176846:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019552.176855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334658688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019552.176865:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019552.176868:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019552.176869:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019552.176873:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3818 00000100:00080000:2.0:1734019552.176876:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019552.176880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019552.176884:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019552.176893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334658816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019552.176902:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019552.176905:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019552.176907:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019552.176910:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3818 00000100:00080000:2.0:1734019552.176913:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019552.176915:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334658432:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019552.176917:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019552.176921:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019552.176930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334658944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019552.176938:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019552.176941:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019552.176942:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019552.176946:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3818 00000100:00080000:2.0:1734019552.176949:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019552.176952:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019552.176956:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019552.176964:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334659072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019552.176972:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019552.176976:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019552.176978:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019552.176982:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3818 00000100:00080000:2.0:1734019552.176985:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019552.176988:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019552.176992:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019552.177000:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334659200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019552.177021:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334658560:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019552.177043:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334658688:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019552.177068:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334658816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019552.177084:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334658944:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019552.177099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334659072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019552.177115:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334659200:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019552.444675:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019553.446711:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019554.448771:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019555.182753:0:1969:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019555.182797:0:1969:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019555.182816:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019555.182851:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019555.182858:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019555.182866:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019555.182873:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019555.182879:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019555.182885:0:1969:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734019555.182959:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019552/real 1734019555] req@ffff8800a78a0a80 x1818247334658816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019555.182998:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334658816/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019555.183014:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019555.183020:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019555.183027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334658816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019555.183039:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019552/real 1734019555] req@ffff8800a78a3b80 x1818247334658944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019555.183058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334658944/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019555.183071:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019555.183075:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019555.183081:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334658944:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019555.183091:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019552/real 1734019555] req@ffff8800a78a0700 x1818247334659072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019555.183166:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334659072/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019555.183181:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019555.183186:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019555.183193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334659072:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019555.183204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019552/real 1734019555] req@ffff8800a78a3480 x1818247334659200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019555.183233:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334659200/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019555.183246:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019555.183249:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019555.183254:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334659200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019555.183265:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019552/real 1734019555] req@ffff8800a78a1f80 x1818247334658432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019555.183283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334658432/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019555.183295:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019555.183299:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019555.183304:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334658432:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019555.183314:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019552/real 1734019555] req@ffff8800a78a2680 x1818247334658560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019555.183333:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334658560/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019555.183347:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019555.183351:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019555.183356:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334658560:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019555.183367:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019552/real 1734019555] req@ffff8800a78a1880 x1818247334658688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019555.183384:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334658688/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019555.183398:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019555.183402:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019555.183407:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334658688:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019555.450739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019556.452730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019557.184538:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019557.184549:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019557.184553:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019557.184564:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3823 00000100:00080000:2.0:1734019557.184573:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019557.184581:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019557.184618:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334659328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019557.184651:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019557.184655:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019557.184657:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019557.184661:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3823 00000100:00080000:2.0:1734019557.184664:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019557.184668:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019557.184678:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019557.184689:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334659456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019557.184700:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019557.184703:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019557.184705:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019557.184708:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3823 00000100:00080000:2.0:1734019557.184711:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019557.184714:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019557.184719:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019557.184727:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334659584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019557.184736:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019557.184739:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019557.184740:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019557.184744:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3823 00000100:00080000:2.0:1734019557.184746:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019557.184749:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019557.184753:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019557.184760:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334659712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019557.184767:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334659328:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019557.184769:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019557.184772:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019557.184774:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019557.184778:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3823 00000100:00080000:2.0:1734019557.184781:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019557.184784:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019557.184788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019557.184795:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334659840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019557.184803:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019557.184806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019557.184808:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019557.184811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3823 00000100:00080000:2.0:1734019557.184813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019557.184816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019557.184820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019557.184827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334659968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019557.184832:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019557.184835:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019557.184838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019557.184839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019557.184843:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3823 00000100:00100000:1.0:1734019557.184853:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019557/real 1734019557] req@ffff8800a78a1880 x1818247334659328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019557.184854:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019557.184858:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019557.184862:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019557.184870:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334660096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019557.184872:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334659328/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019557.184883:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019557.184886:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019557.184890:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334659328:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019557.184898:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334659456:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019557.184914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019557.184921:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019557/real 1734019557] req@ffff8800a78a2680 x1818247334659456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019557.184932:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334659456/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019557.184941:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019557.184944:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019557.184947:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334659456:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019557.184953:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334659584:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019557.184967:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019557.184973:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019557/real 1734019557] req@ffff8800a78a1f80 x1818247334659584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019557.184983:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334659584/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019557.184990:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019557.184993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019557.184996:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334659584:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019557.185020:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334659712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019557.185033:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019557.185040:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019557/real 1734019557] req@ffff8800a78a3480 x1818247334659712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019557.185050:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334659712/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019557.185057:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019557.185060:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019557.185063:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334659712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019557.185069:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334659840:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019557.185116:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019557.185124:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019557/real 1734019557] req@ffff8800a78a0700 x1818247334659840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019557.185135:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334659840/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019557.185143:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019557.185145:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019557.185148:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334659840:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019557.185154:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334659968:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019557.185166:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019557.185172:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019557/real 1734019557] req@ffff8800a78a3b80 x1818247334659968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019557.185182:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334659968/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019557.185190:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019557.185192:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019557.185195:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334659968:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019557.185201:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334660096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019557.185213:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019557.185219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019557/real 1734019557] req@ffff8800a78a0a80 x1818247334660096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019557.185230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334660096/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019557.185237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019557.185239:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019557.185242:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334660096:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019557.454757:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019558.456752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019559.458732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019560.460721:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019561.462727:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019562.192605:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019562.192615:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019562.192619:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019562.192629:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3828 00000100:00080000:2.0:1734019562.192638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019562.192646:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019562.192682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334660224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019562.192716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019562.192720:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019562.192722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019562.192727:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3828 00000100:00080000:2.0:1734019562.192730:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019562.192734:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019562.192743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019562.192754:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334660352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019562.192764:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019562.192767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019562.192769:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019562.192774:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3828 00000100:00080000:2.0:1734019562.192776:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019562.192780:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019562.192785:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019562.192793:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334660480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019562.192802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019562.192806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019562.192807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019562.192810:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3828 00000100:00080000:2.0:1734019562.192813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019562.192816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019562.192819:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334660224:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019562.192820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019562.192828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334660608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019562.192837:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019562.192840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019562.192842:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019562.192845:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3828 00000100:00080000:2.0:1734019562.192848:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019562.192851:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019562.192855:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019562.192862:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334660736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019562.192870:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019562.192873:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019562.192875:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019562.192879:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3828 00000100:00080000:2.0:1734019562.192881:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019562.192883:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019562.192885:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019562.192889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019562.192896:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334660864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019562.192903:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019562/real 1734019562] req@ffff8800a78a0a80 x1818247334660224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019562.192904:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019562.192907:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019562.192909:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019562.192912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3828 00000100:00080000:2.0:1734019562.192915:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019562.192918:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019562.192922:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019562.192923:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334660224/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019562.192929:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334660992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019562.192933:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019562.192936:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019562.192941:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334660224:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019562.192949:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334660352:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019562.192965:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019562.192972:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019562/real 1734019562] req@ffff8800a78a3b80 x1818247334660352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019562.192983:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334660352/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019562.193021:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019562.193024:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019562.193028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334660352:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019562.193034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334660480:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019562.193052:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019562.193059:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019562/real 1734019562] req@ffff8800a78a0700 x1818247334660480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019562.193070:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334660480/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019562.193078:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019562.193080:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019562.193083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334660480:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019562.193107:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334660608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019562.193120:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019562.193126:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019562/real 1734019562] req@ffff8800a78a3480 x1818247334660608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019562.193136:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334660608/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019562.193144:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019562.193146:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019562.193149:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334660608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019562.193156:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334660736:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019562.193169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019562.193175:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019562/real 1734019562] req@ffff8800a78a1f80 x1818247334660736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019562.193186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334660736/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019562.193193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019562.193195:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019562.193198:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334660736:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019562.193203:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334660864:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019562.193215:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019562.193221:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019562/real 1734019562] req@ffff8800a78a2680 x1818247334660864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019562.193231:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334660864/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019562.193238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019562.193240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019562.193243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334660864:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019562.193248:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334660992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019562.193260:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019562.193266:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019562/real 1734019562] req@ffff8800a78a1880 x1818247334660992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019562.193277:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334660992/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019562.193283:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019562.193285:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019562.193288:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334660992:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019562.464730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019563.466767:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019564.468719:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019565.470724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019566.472745:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019567.200597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019567.200607:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019567.200612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019567.200622:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3833 00000100:00080000:2.0:1734019567.200631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019567.200639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019567.200675:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334661120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019567.200716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019567.200720:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019567.200722:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019567.200726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3833 00000100:00080000:2.0:1734019567.200729:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019567.200733:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019567.200742:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019567.200754:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334661248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019567.200763:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019567.200766:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019567.200768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019567.200771:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3833 00000100:00080000:2.0:1734019567.200774:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019567.200777:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019567.200782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019567.200790:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334661376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019567.200799:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019567.200802:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019567.200803:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019567.200807:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3833 00000100:00080000:2.0:1734019567.200809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019567.200813:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019567.200817:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019567.200825:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334661120:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019567.200825:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334661504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019567.200834:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019567.200837:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019567.200839:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019567.200842:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3833 00000100:00080000:2.0:1734019567.200844:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019567.200847:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019567.200851:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019567.200858:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334661632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019567.200866:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019567.200869:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019567.200871:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019567.200874:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3833 00000100:00080000:2.0:1734019567.200877:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019567.200880:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019567.200883:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019567.200891:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019567.200891:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334661760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019567.200899:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019567.200902:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019567.200904:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019567.200959:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3833 00000100:00080000:2.0:1734019567.200962:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019567.200966:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019567.200970:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019567.200979:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334661888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019567.201117:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019567/real 1734019567] req@ffff8800a78a1880 x1818247334661120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019567.201131:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334661120/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019567.201137:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019567.201139:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019567.201142:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334661120:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019567.201147:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334661248:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019567.201162:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019567.201167:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019567/real 1734019567] req@ffff8800a78a2680 x1818247334661248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019567.201174:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334661248/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019567.201180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019567.201181:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019567.201183:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334661248:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019567.201187:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334661376:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019567.201195:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019567.201199:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019567/real 1734019567] req@ffff8800a78a1f80 x1818247334661376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019567.201225:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334661376/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019567.201233:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019567.201235:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019567.201239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334661376:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019567.201264:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334661504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019567.201279:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019567.201285:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019567/real 1734019567] req@ffff8800a78a3480 x1818247334661504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019567.201296:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334661504/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019567.201303:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019567.201305:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019567.201308:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334661504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019567.201320:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334661632:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019567.201333:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019567.201339:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019567/real 1734019567] req@ffff8800a78a0700 x1818247334661632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019567.201349:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334661632/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019567.201356:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019567.201358:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019567.201361:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334661632:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019567.201366:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334661760:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019567.201378:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019567.201384:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019567/real 1734019567] req@ffff8800a78a3b80 x1818247334661760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019567.201394:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334661760/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019567.201402:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019567.201404:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019567.201407:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334661760:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019567.201412:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334661888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019567.201425:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019567.201431:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019567/real 1734019567] req@ffff8800a78a0a80 x1818247334661888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019567.201441:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334661888/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019567.201448:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019567.201450:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019567.201453:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334661888:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019567.474732:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019568.476714:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019569.478788:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019570.480768:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019571.482738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019572.208592:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019572.208602:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019572.208606:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019572.208617:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3838 00000100:00080000:2.0:1734019572.208627:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019572.208635:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019572.208673:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334662016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019572.208709:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019572.208713:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019572.208715:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019572.208719:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3838 00000100:00080000:2.0:1734019572.208722:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019572.208726:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019572.208737:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019572.208748:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334662144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019572.208759:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019572.208762:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019572.208764:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019572.208767:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3838 00000100:00080000:2.0:1734019572.208770:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019572.208773:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019572.208777:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019572.208786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334662272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019572.208794:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019572.208797:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019572.208799:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019572.208802:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3838 00000100:00080000:2.0:1734019572.208805:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019572.208808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019572.208812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019572.208820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334662400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019572.208852:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334662016:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019572.208861:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019572.208864:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019572.208866:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019572.208870:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3838 00000100:00080000:2.0:1734019572.208873:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019572.208877:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019572.208881:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019572.208890:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334662528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019572.208898:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019572.208901:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019572.208903:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019572.208906:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3838 00000100:00080000:2.0:1734019572.208909:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019572.208912:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019572.208916:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019572.208919:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019572.208924:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334662656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019572.208932:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019572.208935:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019572.208936:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019572.208940:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3838 00000100:00100000:1.0:1734019572.208941:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019572/real 1734019572] req@ffff8800a78a0a80 x1818247334662016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019572.208942:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019572.208945:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019572.208949:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019572.208957:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334662784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019572.208960:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334662016/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019572.208971:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019572.208974:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019572.208979:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334662016:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019572.208986:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334662144:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019572.209003:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019572.209010:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019572/real 1734019572] req@ffff8800a78a3b80 x1818247334662144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019572.209021:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334662144/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019572.209030:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019572.209033:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019572.209036:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334662144:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019572.209042:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334662272:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019572.209055:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019572.209061:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019572/real 1734019572] req@ffff8800a78a0700 x1818247334662272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019572.209071:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334662272/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019572.209079:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019572.209081:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019572.209084:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334662272:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019572.209109:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334662400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019572.209121:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019572.209128:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019572/real 1734019572] req@ffff8800a78a3480 x1818247334662400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019572.209138:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334662400/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019572.209145:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019572.209148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019572.209150:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334662400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019572.209157:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334662528:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019572.209170:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019572.209176:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019572/real 1734019572] req@ffff8800a78a1f80 x1818247334662528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019572.209186:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334662528/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019572.209193:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019572.209195:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019572.209198:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334662528:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019572.209203:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334662656:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019572.209214:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019572.209220:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019572/real 1734019572] req@ffff8800a78a2680 x1818247334662656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019572.209230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334662656/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019572.209238:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019572.209240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019572.209243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334662656:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019572.209249:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334662784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019572.209261:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019572.209267:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019572/real 1734019572] req@ffff8800a78a1880 x1818247334662784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019572.209278:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334662784/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019572.209286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019572.209288:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019572.209291:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334662784:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019572.484728:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019573.486771:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019574.488718:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019575.490715:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019576.492793:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019577.216583:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019577.216596:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019577.216602:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019577.216615:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3843 00000100:00080000:2.0:1734019577.216626:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019577.216636:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019577.216686:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334662912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019577.216726:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019577.216730:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019577.216732:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019577.216738:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3843 00000100:00080000:2.0:1734019577.216766:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019577.216771:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019577.216782:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019577.216797:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334663040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019577.216808:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019577.216811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019577.216813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019577.216817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3843 00000100:00080000:2.0:1734019577.216820:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019577.216823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019577.216829:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019577.216839:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334663168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019577.216850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019577.216853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734019577.216855:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334662912:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019577.216855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019577.216858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3843 00000100:00080000:2.0:1734019577.216861:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019577.216865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019577.216869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019577.216879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334663296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019577.216888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019577.216892:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019577.216893:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019577.216897:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3843 00000100:00080000:2.0:1734019577.216899:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019577.216903:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019577.216907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019577.216915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334663424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019577.216923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019577.216926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019577.216928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019577.216932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3843 00000800:00000100:1.0:1734019577.216934:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019577.216936:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019577.216940:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019577.216944:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019577.216952:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334663552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019577.216959:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019577/real 1734019577] req@ffff8800a78a1880 x1818247334662912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019577.216961:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019577.216964:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019577.216965:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019577.216969:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3843 00000100:00080000:2.0:1734019577.216972:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019577.216975:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019577.216979:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334662912/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019577.216979:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019577.216987:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334663680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019577.216990:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019577.216995:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019577.217000:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334662912:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019577.217009:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334663040:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019577.217027:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019577.217034:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019577/real 1734019577] req@ffff8800a78a2680 x1818247334663040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019577.217046:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334663040/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019577.217055:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019577.217058:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019577.217062:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334663040:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019577.217089:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334663168:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019577.217103:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019577.217110:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019577/real 1734019577] req@ffff8800a78a1f80 x1818247334663168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019577.217121:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334663168/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019577.217129:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019577.217131:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019577.217134:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334663168:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019577.217141:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334663296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019577.217154:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019577.217160:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019577/real 1734019577] req@ffff8800a78a3480 x1818247334663296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019577.217170:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334663296/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019577.217177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019577.217180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019577.217183:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334663296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019577.217190:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334663424:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019577.217204:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019577.217210:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019577/real 1734019577] req@ffff8800a78a0700 x1818247334663424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019577.217220:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334663424/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019577.217228:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019577.217230:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019577.217233:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334663424:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019577.217240:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334663552:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019577.217252:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019577.217258:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019577/real 1734019577] req@ffff8800a78a3b80 x1818247334663552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019577.217268:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334663552/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019577.217276:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019577.217279:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019577.217282:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334663552:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019577.217288:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334663680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019577.217301:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019577.217307:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019577/real 1734019577] req@ffff8800a78a0a80 x1818247334663680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019577.217317:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334663680/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019577.217325:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019577.217327:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019577.217330:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334663680:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019577.494728:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019578.496780:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019579.498786:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019580.500761:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019581.502759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019582.224606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019582.224620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019582.224626:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019582.224640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3848 00000100:00080000:2.0:1734019582.224651:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019582.224706:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019582.224758:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334663808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019582.224806:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019582.224811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019582.224813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019582.224818:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3848 00000100:00080000:2.0:1734019582.224822:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019582.224826:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019582.224839:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019582.224852:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334663936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019582.224862:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019582.224865:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019582.224867:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019582.224871:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3848 00000100:00080000:2.0:1734019582.224873:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019582.224878:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019582.224882:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019582.224892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334664064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019582.224901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019582.224904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019582.224906:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019582.224910:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3848 00000100:00080000:2.0:1734019582.224912:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019582.224916:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019582.224921:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019582.224930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334664192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019582.224940:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019582.224942:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334663808:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019582.224943:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019582.224944:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019582.224948:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3848 00000100:00080000:2.0:1734019582.224950:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019582.224954:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019582.224958:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019582.224967:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334664320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019582.224974:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019582.224977:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019582.224979:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019582.224982:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3848 00000100:00080000:2.0:1734019582.224985:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019582.224988:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019582.224993:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019582.225001:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334664448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019582.225009:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019582.225013:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019582.225015:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734019582.225018:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019582.225019:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3848 00000100:00080000:2.0:1734019582.225022:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019582.225026:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019582.225030:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019582.225038:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334664576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019582.225042:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019582/real 1734019582] req@ffff8800a78a0a80 x1818247334663808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019582.225062:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334663808/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019582.225074:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019582.225077:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019582.225082:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334663808:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019582.225091:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334663936:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019582.225109:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019582.225116:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019582/real 1734019582] req@ffff8800a78a3b80 x1818247334663936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019582.225127:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334663936/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019582.225136:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019582.225140:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019582.225143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334663936:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019582.225150:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334664064:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019582.225164:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019582.225170:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019582/real 1734019582] req@ffff8800a78a0700 x1818247334664064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019582.225180:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334664064/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019582.225187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019582.225190:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019582.225193:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334664064:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019582.225220:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334664192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019582.225234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019582.225240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019582/real 1734019582] req@ffff8800a78a3480 x1818247334664192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019582.225251:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334664192/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019582.225259:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019582.225261:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019582.225264:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334664192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019582.225272:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334664320:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019582.225285:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019582.225291:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019582/real 1734019582] req@ffff8800a78a1f80 x1818247334664320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019582.225301:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334664320/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019582.225308:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019582.225311:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019582.225314:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334664320:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019582.225319:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334664448:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019582.225331:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019582.225337:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019582/real 1734019582] req@ffff8800a78a2680 x1818247334664448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019582.225347:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334664448/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019582.225354:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019582.225356:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019582.225359:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334664448:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019582.225366:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334664576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019582.225379:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019582.225385:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019582/real 1734019582] req@ffff8800a78a1880 x1818247334664576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019582.225396:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334664576/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019582.225404:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019582.225406:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019582.225409:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334664576:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019582.504729:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019583.506733:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019584.508770:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019585.510790:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019586.512769:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019587.232629:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019587.232642:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019587.232647:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019587.232660:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3853 00000100:00080000:2.0:1734019587.232671:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019587.232680:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019587.232731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334664704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019587.232766:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019587.232771:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019587.232773:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019587.232778:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3853 00000100:00080000:2.0:1734019587.232781:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019587.232785:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019587.232797:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019587.232809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334664832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019587.232820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019587.232824:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019587.232825:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019587.232829:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3853 00000100:00080000:2.0:1734019587.232832:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019587.232836:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019587.232841:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019587.232851:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334664960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019587.232859:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019587.232863:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019587.232865:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019587.232868:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3853 00000100:00080000:2.0:1734019587.232870:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019587.232874:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019587.232878:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019587.232884:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334664704:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019587.232887:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334665088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019587.232897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019587.232900:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019587.232902:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019587.232906:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3853 00000100:00080000:2.0:1734019587.232909:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019587.232912:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019587.232917:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019587.232925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334665216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019587.232933:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019587.232936:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019587.232938:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019587.232941:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3853 00000100:00080000:2.0:1734019587.232944:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019587.232947:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019587.232952:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019587.232960:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334665344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019587.232967:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019587.232971:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019587.232972:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019587.232975:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3853 00000100:00080000:2.0:1734019587.232978:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019587.232982:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019587.232986:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019587.232987:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019587.232994:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334665472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019587.233015:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019587/real 1734019587] req@ffff8800a78a1880 x1818247334664704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019587.233036:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334664704/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019587.233048:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019587.233051:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019587.233057:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334664704:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019587.233065:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334664832:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019587.233086:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019587.233093:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019587/real 1734019587] req@ffff8800a78a2680 x1818247334664832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019587.233104:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334664832/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019587.233113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019587.233116:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019587.233119:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334664832:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019587.233126:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334664960:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019587.233140:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019587.233146:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019587/real 1734019587] req@ffff8800a78a1f80 x1818247334664960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019587.233157:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334664960/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019587.233164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019587.233166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019587.233170:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334664960:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019587.233199:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334665088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019587.233213:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019587.233219:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019587/real 1734019587] req@ffff8800a78a3480 x1818247334665088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019587.233230:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334665088/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019587.233237:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019587.233240:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019587.233243:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334665088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019587.233250:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334665216:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019587.233263:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019587.233269:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019587/real 1734019587] req@ffff8800a78a0700 x1818247334665216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019587.233279:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334665216/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019587.233287:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019587.233289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019587.233292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334665216:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019587.233297:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334665344:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019587.233309:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019587.233315:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019587/real 1734019587] req@ffff8800a78a3b80 x1818247334665344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019587.233325:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334665344/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019587.233332:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019587.233335:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019587.233337:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334665344:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019587.233343:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334665472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019587.233355:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019587.233361:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019587/real 1734019587] req@ffff8800a78a0a80 x1818247334665472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019587.233373:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334665472/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019587.233381:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019587.233383:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019587.233386:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334665472:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019587.514828:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019588.516777:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019589.518761:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019590.520682:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019591.522738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019592.240608:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019592.240620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019592.240626:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019592.240640:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3858 00000100:00080000:2.0:1734019592.240651:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019592.240661:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019592.240710:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334665600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019592.240749:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019592.240754:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019592.240756:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019592.240761:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3858 00000100:00080000:2.0:1734019592.240764:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019592.240768:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019592.240780:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019592.240792:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334665728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019592.240802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019592.240805:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019592.240807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019592.240811:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3858 00000100:00080000:2.0:1734019592.240814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019592.240817:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019592.240824:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019592.240834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334665856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019592.240844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019592.240847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019592.240849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019592.240853:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3858 00000100:00080000:2.0:1734019592.240855:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019592.240858:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019592.240863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019592.240871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334665984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019592.240881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019592.240884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019592.240885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019592.240889:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3858 00000100:00100000:1.0:1734019592.240891:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334665600:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019592.240891:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019592.240895:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019592.240899:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019592.240908:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334666112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019592.240916:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019592.240920:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019592.240922:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019592.240926:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3858 00000100:00080000:2.0:1734019592.240929:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019592.240932:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019592.240937:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019592.240945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334666240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019592.240953:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019592.240956:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019592.240957:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019592.240961:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3858 00000100:00080000:2.0:1734019592.240964:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019592.240967:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019592.240968:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019592.240972:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019592.240980:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334666368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019592.240993:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019592/real 1734019592] req@ffff8800a78a0a80 x1818247334665600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019592.241013:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334665600/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019592.241025:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019592.241028:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019592.241033:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334665600:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019592.241041:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334665728:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019592.241059:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019592.241066:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019592/real 1734019592] req@ffff8800a78a3b80 x1818247334665728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019592.241077:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334665728/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019592.241086:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019592.241089:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019592.241093:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334665728:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019592.241099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334665856:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019592.241114:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019592.241120:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019592/real 1734019592] req@ffff8800a78a0700 x1818247334665856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019592.241130:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334665856/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019592.241137:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019592.241140:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019592.241143:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334665856:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019592.241171:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334665984:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019592.241185:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019592.241192:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019592/real 1734019592] req@ffff8800a78a3480 x1818247334665984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019592.241203:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334665984/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019592.241210:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019592.241213:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019592.241216:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334665984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019592.241222:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334666112:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019592.241236:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019592.241242:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019592/real 1734019592] req@ffff8800a78a1f80 x1818247334666112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019592.241253:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334666112/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019592.241260:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019592.241263:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019592.241266:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334666112:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019592.241272:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334666240:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019592.241284:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019592.241290:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019592/real 1734019592] req@ffff8800a78a2680 x1818247334666240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019592.241300:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334666240/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019592.241307:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019592.241309:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019592.241312:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334666240:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019592.241318:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334666368:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019592.241331:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019592.241337:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019592/real 1734019592] req@ffff8800a78a1880 x1818247334666368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019592.241348:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334666368/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019592.241355:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019592.241357:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019592.241360:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334666368:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019592.524738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019593.526774:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019594.528762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019595.530738:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019596.532738:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019597.248610:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019597.248618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019597.248623:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019597.248632:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3863 00000100:00080000:2.0:1734019597.248641:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019597.248649:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019597.248685:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334666496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019597.248725:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019597.248729:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019597.248731:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019597.248735:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3863 00000100:00080000:2.0:1734019597.248739:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019597.248742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019597.248751:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019597.248762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334666624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019597.248772:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019597.248775:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019597.248777:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019597.248781:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3863 00000100:00080000:2.0:1734019597.248783:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019597.248787:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019597.248791:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019597.248800:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334666752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019597.248809:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019597.248812:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019597.248813:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019597.248817:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3863 00000100:00080000:2.0:1734019597.248819:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019597.248823:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019597.248827:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019597.248835:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334666880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019597.248837:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334666496:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019597.248844:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019597.248847:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019597.248849:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019597.248852:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3863 00000100:00080000:2.0:1734019597.248854:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019597.248857:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019597.248861:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019597.248868:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334667008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019597.248876:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019597.248879:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019597.248880:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019597.248884:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3863 00000100:00080000:2.0:1734019597.248886:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019597.248889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019597.248893:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019597.248900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334667136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019597.248904:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019597.248908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019597.248912:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019597.248913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019597.248917:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3863 00000100:00080000:2.0:1734019597.248920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019597.248923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019597.248925:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019597/real 1734019597] req@ffff8800a78a1880 x1818247334666496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019597.248927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019597.248934:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334667264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019597.248944:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334666496/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019597.248955:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019597.248958:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019597.248963:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334666496:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019597.248970:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334666624:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019597.248987:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019597.248994:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019597/real 1734019597] req@ffff8800a78a2680 x1818247334666624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019597.249005:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334666624/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019597.249014:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019597.249017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019597.249021:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334666624:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019597.249026:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334666752:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019597.249040:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019597.249046:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019597/real 1734019597] req@ffff8800a78a1f80 x1818247334666752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019597.249056:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334666752/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019597.249063:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019597.249066:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019597.249069:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334666752:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019597.249093:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334666880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019597.249106:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019597.249112:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019597/real 1734019597] req@ffff8800a78a3480 x1818247334666880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019597.249122:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334666880/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019597.249129:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019597.249132:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019597.249135:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334666880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019597.249141:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334667008:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019597.249154:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019597.249160:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019597/real 1734019597] req@ffff8800a78a0700 x1818247334667008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019597.249170:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334667008/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019597.249177:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019597.249179:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019597.249182:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334667008:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019597.249187:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334667136:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019597.249198:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019597.249204:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019597/real 1734019597] req@ffff8800a78a3b80 x1818247334667136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019597.249214:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334667136/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019597.249221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019597.249223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019597.249226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334667136:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019597.249232:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334667264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019597.249245:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019597.249251:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019597/real 1734019597] req@ffff8800a78a0a80 x1818247334667264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019597.249261:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334667264/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019597.249268:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019597.249270:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019597.249273:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334667264:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019597.534729:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019598.536764:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019599.538763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019600.540750:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019601.542707:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019602.256606:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019602.256616:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019602.256620:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019602.256630:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3868 00000100:00080000:2.0:1734019602.256638:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019602.256646:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019602.256684:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334667392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019602.256718:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019602.256722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019602.256724:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019602.256728:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3868 00000100:00080000:2.0:1734019602.256731:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019602.256735:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019602.256745:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019602.256756:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334667520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019602.256767:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019602.256770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019602.256771:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019602.256775:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3868 00000100:00080000:2.0:1734019602.256777:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019602.256781:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019602.256786:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019602.256794:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334667648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019602.256802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019602.256806:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019602.256807:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019602.256810:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3868 00000100:00080000:2.0:1734019602.256813:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019602.256816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019602.256820:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019602.256828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334667776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019602.256836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019602.256838:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334667392:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019602.256839:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019602.256841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019602.256844:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3868 00000100:00080000:2.0:1734019602.256847:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019602.256850:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019602.256854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019602.256861:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334667904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019602.256869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019602.256872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019602.256873:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019602.256877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3868 00000100:00080000:2.0:1734019602.256879:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019602.256882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019602.256886:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019602.256893:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334668032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019602.256901:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019602.256904:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019602.256905:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734019602.256906:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019602.256909:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3868 00000100:00080000:2.0:1734019602.256911:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019602.256914:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019602.256918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019602.256925:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334668160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019602.256928:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019602/real 1734019602] req@ffff8800a78a0a80 x1818247334667392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019602.256947:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334667392/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019602.256958:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019602.256961:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019602.256966:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334667392:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019602.256973:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334667520:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019602.256990:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019602.256997:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019602/real 1734019602] req@ffff8800a78a3b80 x1818247334667520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019602.257008:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334667520/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019602.257017:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019602.257020:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019602.257023:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334667520:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019602.257029:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334667648:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019602.257042:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019602.257048:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019602/real 1734019602] req@ffff8800a78a0700 x1818247334667648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019602.257058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334667648/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019602.257065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019602.257068:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019602.257071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334667648:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019602.257095:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334667776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019602.257108:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019602.257114:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019602/real 1734019602] req@ffff8800a78a3480 x1818247334667776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019602.257125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334667776/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019602.257132:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019602.257134:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019602.257137:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334667776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019602.257144:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334667904:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019602.257156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019602.257162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019602/real 1734019602] req@ffff8800a78a1f80 x1818247334667904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019602.257172:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334667904/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019602.257180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019602.257182:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019602.257185:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334667904:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019602.257191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334668032:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019602.257202:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019602.257209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019602/real 1734019602] req@ffff8800a78a2680 x1818247334668032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019602.257219:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334668032/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019602.257226:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019602.257228:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019602.257231:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334668032:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019602.257236:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334668160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019602.257249:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019602.257255:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019602/real 1734019602] req@ffff8800a78a1880 x1818247334668160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019602.257265:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334668160/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019602.257273:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019602.257275:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019602.257278:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334668160:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019602.544739:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019603.546770:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019604.548749:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019605.550737:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019606.552748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019607.264581:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019607.264596:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019607.264602:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019607.264615:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3873 00000100:00080000:2.0:1734019607.264626:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019607.264635:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019607.264686:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334668288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019607.264725:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019607.264730:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019607.264732:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019607.264737:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3873 00000100:00080000:2.0:1734019607.264740:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019607.264744:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019607.264755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019607.264768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334668416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019607.264778:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019607.264781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019607.264783:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019607.264787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3873 00000100:00080000:2.0:1734019607.264790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019607.264794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019607.264800:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019607.264809:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334668544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019607.264819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019607.264823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019607.264824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019607.264828:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3873 00000100:00080000:2.0:1734019607.264830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019607.264834:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019607.264838:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019607.264846:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334668672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019607.264856:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019607.264859:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019607.264860:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019607.264864:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3873 00000100:00100000:1.0:1734019607.264865:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334668288:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019607.264867:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019607.264870:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019607.264874:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019607.264883:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334668800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019607.264892:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019607.264895:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019607.264897:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019607.264901:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3873 00000100:00080000:2.0:1734019607.264904:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019607.264908:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019607.264912:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019607.264920:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334668928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019607.264928:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019607.264931:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019607.264933:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019607.264936:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3873 00000100:00080000:2.0:1734019607.264940:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019607.264941:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019607.264943:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019607.264948:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019607.264955:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334669056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019607.264963:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019607/real 1734019607] req@ffff8800a78a1880 x1818247334668288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019607.264984:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334668288/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019607.264996:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019607.265000:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019607.265005:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334668288:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019607.265013:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334668416:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019607.265031:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019607.265038:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019607/real 1734019607] req@ffff8800a78a2680 x1818247334668416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019607.265049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334668416/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019607.265058:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019607.265061:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019607.265065:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334668416:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019607.265071:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334668544:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019607.265085:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019607.265091:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019607/real 1734019607] req@ffff8800a78a1f80 x1818247334668544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019607.265101:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334668544/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019607.265109:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019607.265111:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019607.265115:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334668544:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019607.265142:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334668672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019607.265156:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019607.265162:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019607/real 1734019607] req@ffff8800a78a3480 x1818247334668672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019607.265173:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334668672/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019607.265180:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019607.265183:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019607.265186:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334668672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019607.265193:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334668800:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019607.265206:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019607.265212:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019607/real 1734019607] req@ffff8800a78a0700 x1818247334668800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019607.265222:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334668800/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019607.265229:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019607.265231:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019607.265268:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334668800:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019607.265274:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334668928:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019607.265291:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019607.265298:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019607/real 1734019607] req@ffff8800a78a3b80 x1818247334668928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019607.265308:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334668928/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019607.265315:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019607.265318:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019607.265321:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334668928:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019607.265327:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334669056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019607.265340:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019607.265346:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019607/real 1734019607] req@ffff8800a78a0a80 x1818247334669056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019607.265357:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334669056/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019607.265364:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019607.265367:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019607.265370:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334669056:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019607.554739:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019608.556754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019609.558743:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019610.560668:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019611.562667:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019612.272604:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019612.272618:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019612.272624:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019612.272637:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3878 00000100:00080000:2.0:1734019612.272648:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019612.272658:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019612.272709:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334669184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019612.272758:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019612.272762:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019612.272764:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019612.272769:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3878 00000100:00080000:2.0:1734019612.272773:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019612.272777:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019612.272788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019612.272801:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334669312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019612.272811:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019612.272814:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019612.272816:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019612.272820:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3878 00000100:00080000:2.0:1734019612.272823:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019612.272827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019612.272831:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019612.272841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334669440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019612.272850:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019612.272853:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019612.272855:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019612.272858:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3878 00000100:00080000:2.0:1734019612.272861:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019612.272865:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019612.272869:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019612.272878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334669568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019612.272888:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019612.272891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019612.272892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019612.272896:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3878 00000100:00080000:2.0:1734019612.272899:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019612.272900:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334669184:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019612.272902:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019612.272907:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019612.272915:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334669696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019612.272923:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019612.272926:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019612.272928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019612.272932:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3878 00000100:00080000:2.0:1734019612.272934:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019612.272937:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019612.272942:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019612.272950:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334669824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019612.272958:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019612.272962:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019612.272964:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019612.272967:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3878 00000800:00000100:1.0:1734019612.272971:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019612.272971:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019612.272975:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019612.272979:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019612.272986:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334669952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019612.272993:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019612/real 1734019612] req@ffff8800a78a0a80 x1818247334669184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019612.273014:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334669184/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019612.273026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019612.273029:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019612.273034:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334669184:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019612.273042:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334669312:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019612.273059:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019612.273066:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019612/real 1734019612] req@ffff8800a78a3b80 x1818247334669312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019612.273077:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334669312/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019612.273086:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019612.273089:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019612.273093:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334669312:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019612.273099:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334669440:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019612.273112:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019612.273119:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019612/real 1734019612] req@ffff8800a78a0700 x1818247334669440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019612.273129:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334669440/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019612.273137:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019612.273139:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019612.273142:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334669440:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019612.273216:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334669568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019612.273234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019612.273242:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019612/real 1734019612] req@ffff8800a78a3480 x1818247334669568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019612.273252:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334669568/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019612.273261:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019612.273263:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019612.273266:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334669568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019612.273273:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334669696:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019612.273287:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019612.273293:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019612/real 1734019612] req@ffff8800a78a1f80 x1818247334669696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019612.273304:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334669696/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019612.273311:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019612.273314:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019612.273317:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334669696:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019612.273323:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334669824:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019612.273334:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019612.273340:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019612/real 1734019612] req@ffff8800a78a2680 x1818247334669824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019612.273350:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334669824/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019612.273357:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019612.273359:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019612.273362:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334669824:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019612.273368:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334669952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019612.273381:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019612.273387:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019612/real 1734019612] req@ffff8800a78a1880 x1818247334669952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019612.273398:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334669952/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019612.273405:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019612.273407:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019612.273410:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334669952:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019612.564731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019613.566731:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019614.568717:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019615.570774:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019616.572742:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019617.280713:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019617.280723:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019617.280727:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019617.280738:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3883 00000100:00080000:0.0:1734019617.280747:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019617.280755:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:0.0:1734019617.280790:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c380 x1818247334670080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019617.280827:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019617.280831:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019617.280833:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019617.280837:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3883 00000100:00080000:0.0:1734019617.280840:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019617.280844:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019617.280854:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019617.280864:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c700 x1818247334670208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019617.280874:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019617.280877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019617.280879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019617.280882:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3883 00000100:00080000:0.0:1734019617.280886:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019617.280889:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019617.280894:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1734019617.280902:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ed80 x1818247334670336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019617.280910:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019617.280913:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019617.280915:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019617.280918:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3883 00000100:00080000:0.0:1734019617.280920:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019617.280923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019617.280927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:0.0:1734019617.280935:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ca80 x1818247334670464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019617.280935:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334670080:192.168.201.116@tcp:250:kworker.0 00000100:00080000:0.0:1734019617.280944:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019617.280947:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019617.280948:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019617.280952:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3883 00000100:00080000:0.0:1734019617.280955:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019617.280958:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019617.280963:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019617.280970:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934ea00 x1818247334670592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019617.280984:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019617.280987:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019617.280989:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019617.280992:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3883 00000100:00080000:0.0:1734019617.280995:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019617.280998:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019617.281002:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019617.281009:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934c000 x1818247334670720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019617.281019:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019617.281023:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019617.281026:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019617.281028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334670208:192.168.201.116@tcp:38:kworker.0 00000100:00080000:0.0:1734019617.281031:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3883 00000100:00080000:0.0:1734019617.281035:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019617.281040:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:0.0:1734019617.281047:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019617.281049:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334670336:192.168.201.116@tcp:8:kworker.0 00000100:00100000:0.0:1734019617.281061:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a934d180 x1818247334670848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019617.281112:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334670464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019617.281132:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334670592:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019617.281146:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334670720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019617.281161:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334670848:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019617.574726:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019618.576735:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019619.578680:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019620.286738:0:1970:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019620.286783:0:1970:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019620.286802:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019620.286837:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019620.286844:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019620.286851:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019620.286858:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019620.286864:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019620.286870:0:1970:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734019620.286961:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019617/real 1734019620] req@ffff8800a934ca80 x1818247334670464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019620.287001:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ca80 x1818247334670464/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019620.287076:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019620.287082:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019620.287090:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334670464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019620.287103:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019617/real 1734019620] req@ffff8800a934ea00 x1818247334670592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019620.287125:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ea00 x1818247334670592/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019620.287137:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019620.287141:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019620.287147:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334670592:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019620.287157:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019617/real 1734019620] req@ffff8800a934c000 x1818247334670720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019620.287175:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c000 x1818247334670720/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019620.287187:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019620.287191:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019620.287207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334670720:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019620.287217:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019617/real 1734019620] req@ffff8800a934d180 x1818247334670848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019620.287235:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934d180 x1818247334670848/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019620.287257:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019620.287260:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019620.287266:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334670848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019620.287276:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019617/real 1734019620] req@ffff8800a934c380 x1818247334670080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019620.287295:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c380 x1818247334670080/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019620.287307:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019620.287311:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019620.287317:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334670080:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019620.287327:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019617/real 1734019620] req@ffff8800a934c700 x1818247334670208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019620.287345:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934c700 x1818247334670208/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019620.287358:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019620.287361:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019620.287367:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334670208:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019620.287377:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019617/real 1734019620] req@ffff8800a934ed80 x1818247334670336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019620.287395:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a934ed80 x1818247334670336/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019620.287417:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019620.287420:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019620.287425:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a934ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334670336:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019620.580755:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019621.582749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019622.288678:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019622.288688:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019622.288693:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019622.288703:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3888 00000100:00080000:2.0:1734019622.288712:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019622.288719:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019622.288759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334670976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019622.288796:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019622.288801:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019622.288802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019622.288807:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3888 00000100:00080000:2.0:1734019622.288810:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019622.288813:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019622.288823:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019622.288834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334671104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019622.288845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019622.288848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019622.288850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019622.288853:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3888 00000100:00080000:2.0:1734019622.288856:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019622.288859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019622.288863:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019622.288871:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334671232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019622.288880:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019622.288883:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019622.288885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019622.288888:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3888 00000100:00080000:2.0:1734019622.288890:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019622.288893:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019622.288897:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019622.288905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334671360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019622.288912:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334670976:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019622.288914:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019622.288917:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019622.288918:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019622.288921:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3888 00000100:00080000:2.0:1734019622.288924:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019622.288928:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019622.288932:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019622.288939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334671488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019622.288947:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019622.288949:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019622.288951:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019622.288954:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3888 00000100:00080000:2.0:1734019622.288957:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019622.288960:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019622.288964:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019622.288971:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334671616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019622.288978:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019622.288979:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019622.289009:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019622.289010:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019622.289014:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3888 00000100:00080000:2.0:1734019622.289017:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019622.289020:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019622.289024:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019622.289030:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019622/real 1734019622] req@ffff8800a78a1880 x1818247334670976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019622.289033:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334671744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019622.289049:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334670976/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019622.289060:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019622.289063:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019622.289068:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334670976:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019622.289076:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334671104:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019622.289095:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019622.289102:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019622/real 1734019622] req@ffff8800a78a2680 x1818247334671104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019622.289114:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334671104/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019622.289122:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019622.289124:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019622.289128:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334671104:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019622.289134:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334671232:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019622.289148:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019622.289154:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019622/real 1734019622] req@ffff8800a78a1f80 x1818247334671232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019622.289164:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334671232/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019622.289171:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019622.289174:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019622.289177:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334671232:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019622.289201:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334671360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019622.289214:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019622.289221:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019622/real 1734019622] req@ffff8800a78a3480 x1818247334671360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019622.289232:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334671360/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019622.289239:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019622.289241:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019622.289244:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334671360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019622.289250:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334671488:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019622.289262:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019622.289269:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019622/real 1734019622] req@ffff8800a78a0700 x1818247334671488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019622.289279:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334671488/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019622.289286:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019622.289289:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019622.289292:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334671488:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019622.289297:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334671616:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019622.289309:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019622.289315:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019622/real 1734019622] req@ffff8800a78a3b80 x1818247334671616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019622.289324:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334671616/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019622.289331:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019622.289333:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019622.289336:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334671616:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019622.289341:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334671744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019622.289353:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019622.289359:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019622/real 1734019622] req@ffff8800a78a0a80 x1818247334671744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019622.289369:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334671744/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019622.289383:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019622.289385:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019622.289388:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334671744:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019622.584748:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019623.586763:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019624.588714:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019625.590676:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019626.592706:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019627.296545:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019627.296552:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019627.296555:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019627.296562:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3893 00000100:00080000:2.0:1734019627.296569:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019627.296573:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019627.296600:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334671872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019627.296618:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019627.296620:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019627.296621:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019627.296623:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3893 00000100:00080000:2.0:1734019627.296625:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019627.296627:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019627.296633:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019627.296639:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334672000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019627.296644:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019627.296645:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019627.296646:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019627.296649:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3893 00000100:00080000:2.0:1734019627.296650:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019627.296652:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019627.296655:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019627.296660:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334672128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019627.296675:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019627.296676:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019627.296677:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019627.296679:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3893 00000100:00080000:2.0:1734019627.296680:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019627.296682:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019627.296684:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019627.296688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334672256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019627.296692:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019627.296693:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334671872:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019627.296694:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019627.296695:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019627.296697:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3893 00000100:00080000:2.0:1734019627.296698:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019627.296700:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019627.296702:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019627.296706:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334672384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019627.296710:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019627.296711:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019627.296712:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019627.296714:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3893 00000100:00080000:2.0:1734019627.296716:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019627.296718:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019627.296720:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019627.296724:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334672512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019627.296728:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019627.296729:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019627.296730:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019627.296732:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3893 00000100:00080000:2.0:1734019627.296733:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019627.296735:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019627.296736:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019627.296737:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019627.296741:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334672640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019627.296748:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019627/real 1734019627] req@ffff8800a78a0a80 x1818247334671872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019627.296759:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334671872/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019627.296765:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019627.296767:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019627.296769:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334671872:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019627.296773:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334672000:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019627.296783:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019627.296786:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019627/real 1734019627] req@ffff8800a78a3b80 x1818247334672000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019627.296792:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334672000/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019627.296796:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019627.296798:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019627.296800:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334672000:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019627.296803:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334672128:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019627.296811:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019627.296814:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019627/real 1734019627] req@ffff8800a78a0700 x1818247334672128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019627.296819:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334672128/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019627.296823:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019627.296825:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019627.296827:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334672128:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019627.296841:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334672256:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019627.296848:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019627.296851:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019627/real 1734019627] req@ffff8800a78a3480 x1818247334672256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019627.296857:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334672256/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019627.296861:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019627.296862:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019627.296864:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334672256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019627.296866:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334672384:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019627.296874:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019627.296878:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019627/real 1734019627] req@ffff8800a78a1f80 x1818247334672384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019627.296883:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334672384/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019627.296887:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019627.296888:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019627.296889:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334672384:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019627.296892:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334672512:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019627.296938:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019627.296942:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019627/real 1734019627] req@ffff8800a78a2680 x1818247334672512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019627.296948:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334672512/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019627.296952:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019627.296953:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019627.296955:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334672512:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019627.296958:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334672640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019627.296966:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019627.296969:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019627/real 1734019627] req@ffff8800a78a1880 x1818247334672640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019627.296978:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334672640/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019627.296982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019627.296983:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019627.296984:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334672640:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019627.594671:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019628.596637:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019629.598655:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019630.600661:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019631.602605:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019632.304644:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019632.304655:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019632.304660:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019632.304672:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3898 00000100:00080000:2.0:1734019632.304682:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019632.304689:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019632.304731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334672768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019632.304775:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019632.304780:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019632.304781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019632.304786:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3898 00000100:00080000:2.0:1734019632.304789:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019632.304792:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019632.304803:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019632.304859:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334672896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019632.304869:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019632.304872:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019632.304874:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019632.304877:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3898 00000100:00080000:2.0:1734019632.304880:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019632.304884:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019632.304889:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019632.304899:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334673024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019632.304908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019632.304911:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019632.304913:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019632.304915:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334672768:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019632.304917:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3898 00000100:00080000:2.0:1734019632.304919:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019632.304923:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019632.304927:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019632.304936:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334673152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019632.304944:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019632.304947:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019632.304949:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019632.304952:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3898 00000100:00080000:2.0:1734019632.304954:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019632.304957:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019632.304961:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019632.304968:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334673280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019632.304976:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019632.304979:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019632.304981:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734019632.304984:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019632.304984:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3898 00000100:00080000:2.0:1734019632.304987:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019632.304990:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019632.304994:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019632.305002:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334673408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019632.305007:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019632/real 1734019632] req@ffff8800a78a1880 x1818247334672768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019632.305010:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019632.305014:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019632.305015:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019632.305019:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3898 00000100:00080000:2.0:1734019632.305023:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019632.305026:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019632.305027:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334672768/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019632.305030:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019632.305037:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334673536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019632.305038:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019632.305041:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019632.305046:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334672768:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019632.305053:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334672896:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019632.305071:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019632.305077:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019632/real 1734019632] req@ffff8800a78a2680 x1818247334672896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019632.305088:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334672896/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019632.305097:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019632.305100:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019632.305103:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334672896:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019632.305125:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334673024:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019632.305139:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019632.305146:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019632/real 1734019632] req@ffff8800a78a1f80 x1818247334673024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019632.305156:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334673024/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019632.305164:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019632.305166:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019632.305169:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334673024:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019632.305175:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334673152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019632.305187:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019632.305193:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019632/real 1734019632] req@ffff8800a78a3480 x1818247334673152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019632.305203:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334673152/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019632.305210:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019632.305212:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019632.305215:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334673152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019632.305222:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334673280:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019632.305234:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019632.305240:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019632/real 1734019632] req@ffff8800a78a0700 x1818247334673280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019632.305250:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334673280/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019632.305257:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019632.305259:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019632.305263:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334673280:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019632.305269:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334673408:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019632.305281:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019632.305287:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019632/real 1734019632] req@ffff8800a78a3b80 x1818247334673408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019632.305296:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334673408/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019632.305303:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019632.305305:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019632.305308:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334673408:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019632.305314:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334673536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019632.305326:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019632.305332:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019632/real 1734019632] req@ffff8800a78a0a80 x1818247334673536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019632.305343:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334673536/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019632.305350:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019632.305352:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019632.305355:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334673536:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019632.604665:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019633.606819:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019634.608731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019635.610750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019636.612730:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019637.312574:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019637.312587:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019637.312591:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019637.312602:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3903 00000100:00080000:2.0:1734019637.312611:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019637.312620:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019637.312658:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334673664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019637.312692:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019637.312696:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019637.312698:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019637.312702:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3903 00000100:00080000:2.0:1734019637.312705:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019637.312709:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019637.312718:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019637.312756:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334673792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019637.312769:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019637.312771:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334673664:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019637.312772:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019637.312773:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019637.312777:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3903 00000100:00080000:2.0:1734019637.312780:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019637.312783:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019637.312788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019637.312798:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334673920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019637.312807:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019637.312811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019637.312812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019637.312815:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3903 00000100:00080000:2.0:1734019637.312818:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019637.312821:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019637.312825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019637.312833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334674048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019637.312839:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019637.312841:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019637.312844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019637.312845:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019637.312849:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3903 00000100:00080000:2.0:1734019637.312852:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019637.312855:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019637.312859:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019637.312861:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019637/real 1734019637] req@ffff8800a78a0a80 x1818247334673664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019637.312867:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334674176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019637.312875:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019637.312877:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019637.312879:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019637.312880:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334673664/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019637.312883:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3903 00000100:00080000:2.0:1734019637.312885:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019637.312888:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00080000:1.0:1734019637.312891:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1734019637.312892:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734019637.312894:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019637.312899:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334673664:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019637.312900:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334674304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019637.312907:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019637.312910:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019637.312912:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019637.312915:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3903 00000100:00080000:2.0:1734019637.312917:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019637.312920:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019637.312922:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334673792:192.168.201.116@tcp:38:kworker.0 00000008:00100000:2.0:1734019637.312925:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019637.312932:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334674432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019637.312940:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019637.312947:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019637/real 1734019637] req@ffff8800a78a3b80 x1818247334673792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019637.312959:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334673792/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019637.312968:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019637.312970:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019637.312974:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334673792:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019637.312980:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334673920:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019637.312993:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019637.312999:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019637/real 1734019637] req@ffff8800a78a0700 x1818247334673920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019637.313009:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334673920/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019637.313017:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019637.313019:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019637.313023:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334673920:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019637.313028:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334674048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019637.313041:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019637.313047:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019637/real 1734019637] req@ffff8800a78a3480 x1818247334674048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019637.313058:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334674048/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019637.313065:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019637.313068:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019637.313071:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334674048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019637.313077:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334674176:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019637.313090:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019637.313096:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019637/real 1734019637] req@ffff8800a78a1f80 x1818247334674176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019637.313106:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334674176/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019637.313113:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019637.313115:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019637.313118:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334674176:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019637.313124:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334674304:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019637.313136:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019637.313142:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019637/real 1734019637] req@ffff8800a78a2680 x1818247334674304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019637.313151:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334674304/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019637.313158:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019637.313161:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019637.313164:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334674304:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019637.313181:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334674432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019637.313194:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019637.313201:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019637/real 1734019637] req@ffff8800a78a1880 x1818247334674432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019637.313212:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334674432/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019637.313219:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019637.313222:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019637.313224:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334674432:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019637.614699:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019638.616752:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019639.618635:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019640.620711:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019641.622653:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019642.320594:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019642.320603:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019642.320606:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019642.320614:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3908 00000100:00080000:2.0:1734019642.320622:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019642.320627:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019642.320682:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334674560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019642.320706:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019642.320708:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019642.320709:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019642.320711:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3908 00000100:00080000:2.0:1734019642.320713:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019642.320715:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019642.320723:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019642.320730:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334674688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019642.320735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019642.320737:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019642.320738:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019642.320740:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3908 00000100:00080000:2.0:1734019642.320741:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019642.320743:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019642.320746:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019642.320752:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334674816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019642.320756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019642.320758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019642.320758:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019642.320760:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3908 00000100:00080000:2.0:1734019642.320762:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019642.320763:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019642.320765:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019642.320769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334674944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019642.320773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019642.320775:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019642.320775:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019642.320778:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3908 00000100:00080000:2.0:1734019642.320779:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019642.320781:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019642.320783:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019642.320787:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334675072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019642.320791:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019642.320792:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019642.320793:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019642.320796:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3908 00000100:00080000:2.0:1734019642.320797:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019642.320799:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019642.320802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019642.320803:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334674560:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019642.320806:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334675200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019642.320810:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019642.320811:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019642.320812:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019642.320813:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3908 00000100:00080000:2.0:1734019642.320814:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019642.320816:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019642.320818:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019642.320822:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334675328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019642.320852:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019642.320866:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019642/real 1734019642] req@ffff8800a78a1880 x1818247334674560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019642.320876:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334674560/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019642.320881:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019642.320883:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019642.320885:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334674560:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019642.320890:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334674688:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019642.320898:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019642.320901:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019642/real 1734019642] req@ffff8800a78a2680 x1818247334674688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019642.320906:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334674688/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019642.320911:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019642.320912:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019642.320914:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334674688:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019642.320917:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334674816:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019642.320923:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019642.320926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019642/real 1734019642] req@ffff8800a78a1f80 x1818247334674816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019642.320930:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334674816/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019642.320933:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019642.320935:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019642.320936:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334674816:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019642.320939:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334674944:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019642.320945:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019642.320947:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019642/real 1734019642] req@ffff8800a78a3480 x1818247334674944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019642.320952:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334674944/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019642.320955:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019642.320956:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019642.320957:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334674944:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019642.320973:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334675072:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019642.320979:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019642.320982:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019642/real 1734019642] req@ffff8800a78a0700 x1818247334675072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019642.320987:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334675072/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019642.320991:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019642.320992:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019642.320993:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334675072:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019642.320996:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334675200:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019642.321001:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019642.321004:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019642/real 1734019642] req@ffff8800a78a3b80 x1818247334675200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019642.321008:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334675200/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019642.321012:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019642.321013:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019642.321014:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334675200:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019642.321017:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334675328:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019642.321023:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019642.321026:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019642/real 1734019642] req@ffff8800a78a0a80 x1818247334675328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019642.321031:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334675328/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019642.321034:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019642.321035:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019642.321036:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334675328:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019642.624690:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019643.626658:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019644.628654:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019645.630659:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019646.632661:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019647.328636:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019647.328644:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019647.328648:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019647.328655:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3913 00000100:00080000:2.0:1734019647.328662:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019647.328667:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019647.328696:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334675456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019647.328724:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019647.328727:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019647.328728:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019647.328730:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3913 00000100:00080000:2.0:1734019647.328732:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019647.328735:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019647.328742:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019647.328748:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334675584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019647.328753:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019647.328755:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019647.328756:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019647.328758:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3913 00000100:00080000:2.0:1734019647.328759:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019647.328762:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019647.328764:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019647.328769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334675712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019647.328773:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019647.328775:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019647.328776:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019647.328778:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3913 00000100:00080000:2.0:1734019647.328779:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019647.328781:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019647.328783:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019647.328788:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334675456:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019647.328796:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334675840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019647.328802:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019647.328803:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019647.328804:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019647.328806:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3913 00000100:00080000:2.0:1734019647.328807:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019647.328809:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019647.328811:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019647.328815:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334675968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019647.328819:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019647.328821:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019647.328822:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019647.328824:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3913 00000100:00080000:2.0:1734019647.328825:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019647.328827:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019647.328830:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019647.328834:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334676096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019647.328838:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019647.328840:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019647.328841:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734019647.328842:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019647.328843:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3913 00000100:00080000:2.0:1734019647.328845:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019647.328847:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019647.328849:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019647.328853:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334676224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019647.328860:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019647/real 1734019647] req@ffff8800a78a0a80 x1818247334675456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019647.328874:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334675456/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019647.328882:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019647.328884:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019647.328888:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334675456:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019647.328893:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334675584:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019647.328906:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019647.328910:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019647/real 1734019647] req@ffff8800a78a3b80 x1818247334675584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019647.328916:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334675584/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019647.328921:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019647.328922:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019647.328924:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334675584:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019647.328927:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334675712:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019647.328934:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019647.328938:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019647/real 1734019647] req@ffff8800a78a0700 x1818247334675712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019647.328943:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334675712/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019647.328946:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019647.328948:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019647.328950:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334675712:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019647.328966:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334675840:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019647.328973:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019647.328976:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019647/real 1734019647] req@ffff8800a78a3480 x1818247334675840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019647.328982:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334675840/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019647.328985:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019647.328987:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019647.328988:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334675840:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019647.328991:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334675968:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019647.328999:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019647.329002:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019647/real 1734019647] req@ffff8800a78a1f80 x1818247334675968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019647.329007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334675968/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019647.329010:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019647.329011:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019647.329013:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334675968:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019647.329015:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334676096:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019647.329022:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019647.329025:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019647/real 1734019647] req@ffff8800a78a2680 x1818247334676096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019647.329029:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334676096/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019647.329033:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019647.329035:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019647.329036:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334676096:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019647.329039:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334676224:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019647.329046:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019647.329049:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019647/real 1734019647] req@ffff8800a78a1880 x1818247334676224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019647.329054:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334676224/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019647.329057:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019647.329058:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019647.329060:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334676224:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019647.634622:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019648.636623:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019649.638635:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019650.640655:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019651.642661:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019652.336569:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019652.336579:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019652.336584:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019652.336594:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3918 00000100:00080000:2.0:1734019652.336605:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019652.336612:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019652.336650:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334676352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019652.336685:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019652.336689:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019652.336692:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019652.336696:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3918 00000100:00080000:2.0:1734019652.336699:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019652.336703:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019652.336712:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019652.336723:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334676480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019652.336735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019652.336738:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019652.336740:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019652.336743:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3918 00000100:00080000:2.0:1734019652.336746:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019652.336750:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019652.336754:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019652.336762:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334676608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019652.336771:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334676352:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019652.336771:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019652.336774:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019652.336776:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019652.336780:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3918 00000100:00080000:2.0:1734019652.336782:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019652.336785:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019652.336789:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019652.336796:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334676736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019652.336805:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019652.336807:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019652.336809:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019652.336812:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3918 00000100:00080000:2.0:1734019652.336815:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019652.336818:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019652.336822:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019652.336828:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334676864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019652.336836:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019652.336838:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019652.336840:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019652.336843:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3918 00000100:00080000:2.0:1734019652.336846:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019652.336849:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019652.336853:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019652.336854:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019652.336860:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334676992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019652.336868:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019652.336871:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019652.336872:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019652.336876:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019652/real 1734019652] req@ffff8800a78a1880 x1818247334676352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019652.336876:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3918 00000100:00080000:2.0:1734019652.336878:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019652.336882:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019652.336885:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019652.336892:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334677120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019652.336904:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334676352/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019652.336915:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019652.336918:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019652.336923:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334676352:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019652.336931:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334676480:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019652.336951:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019652.336958:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019652/real 1734019652] req@ffff8800a78a2680 x1818247334676480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019652.336969:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334676480/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019652.336987:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019652.336989:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019652.336993:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334676480:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019652.337022:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334676608:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019652.337038:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019652.337045:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019652/real 1734019652] req@ffff8800a78a1f80 x1818247334676608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019652.337056:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334676608/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019652.337064:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019652.337067:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019652.337070:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334676608:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019652.337083:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334676736:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019652.337097:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019652.337103:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019652/real 1734019652] req@ffff8800a78a3480 x1818247334676736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019652.337114:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334676736/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019652.337121:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019652.337123:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019652.337127:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334676736:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019652.337133:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334676864:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019652.337146:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019652.337153:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019652/real 1734019652] req@ffff8800a78a0700 x1818247334676864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019652.337163:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334676864/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019652.337178:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019652.337180:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019652.337183:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334676864:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019652.337189:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334676992:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019652.337203:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019652.337209:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019652/real 1734019652] req@ffff8800a78a3b80 x1818247334676992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019652.337227:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334676992/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019652.337235:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019652.337237:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019652.337240:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334676992:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019652.337245:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334677120:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019652.337258:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019652.337272:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019652/real 1734019652] req@ffff8800a78a0a80 x1818247334677120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019652.337283:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334677120/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019652.337291:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019652.337293:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019652.337296:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334677120:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019652.644733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019653.646593:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019654.648615:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019655.650626:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019656.652601:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019657.344570:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019657.344578:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019657.344581:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019657.344588:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3923 00000100:00080000:2.0:1734019657.344594:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019657.344599:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019657.344628:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334677248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019657.344650:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019657.344653:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019657.344654:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019657.344656:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3923 00000100:00080000:2.0:1734019657.344658:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019657.344664:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019657.344671:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019657.344677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334677376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019657.344682:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019657.344684:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019657.344685:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019657.344687:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3923 00000100:00080000:2.0:1734019657.344688:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019657.344690:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019657.344694:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019657.344698:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334677504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019657.344703:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334677248:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019657.344704:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019657.344705:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019657.344706:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019657.344708:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3923 00000100:00080000:2.0:1734019657.344710:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019657.344711:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019657.344714:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019657.344718:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334677632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019657.344722:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019657.344724:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019657.344724:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019657.344726:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3923 00000100:00080000:2.0:1734019657.344727:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019657.344729:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019657.344731:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019657.344735:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334677760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019657.344739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019657.344741:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019657.344742:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019657.344744:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3923 00000100:00080000:2.0:1734019657.344745:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019657.344747:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019657.344749:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019657.344750:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019657.344753:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334677888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019657.344757:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019657.344759:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019657.344760:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019657.344761:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3923 00000100:00080000:2.0:1734019657.344763:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019657.344764:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019657.344765:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019657/real 1734019657] req@ffff8800a78a0a80 x1818247334677248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019657.344767:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019657.344771:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334678016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019657.344778:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334677248/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019657.344786:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019657.344788:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019657.344792:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334677248:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019657.344797:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334677376:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019657.344811:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019657.344816:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019657/real 1734019657] req@ffff8800a78a3b80 x1818247334677376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019657.344824:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334677376/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019657.344830:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019657.344832:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019657.344835:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334677376:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019657.344853:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334677504:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019657.344864:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019657.344869:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019657/real 1734019657] req@ffff8800a78a0700 x1818247334677504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019657.344877:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334677504/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019657.344883:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019657.344884:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019657.344887:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334677504:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019657.344891:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334677632:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019657.344901:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019657.344906:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019657/real 1734019657] req@ffff8800a78a3480 x1818247334677632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019657.344913:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334677632/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019657.344918:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019657.344920:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019657.344923:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334677632:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019657.344927:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334677760:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019657.344938:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019657.344943:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019657/real 1734019657] req@ffff8800a78a1f80 x1818247334677760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019657.344951:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334677760/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019657.344956:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019657.344957:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019657.344960:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334677760:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019657.344964:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334677888:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019657.344973:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019657.344978:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019657/real 1734019657] req@ffff8800a78a2680 x1818247334677888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019657.344985:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334677888/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019657.344991:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019657.344992:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019657.344994:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334677888:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019657.344999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334678016:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019657.345009:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019657.345014:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019657/real 1734019657] req@ffff8800a78a1880 x1818247334678016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019657.345022:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334678016/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019657.345027:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019657.345029:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019657.345031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334678016:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019657.654600:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019658.656626:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019659.658664:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019660.660667:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019661.662681:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019662.352577:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019662.352585:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019662.352588:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019662.352595:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3928 00000100:00080000:2.0:1734019662.352601:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019662.352606:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019662.352643:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334678144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019662.352669:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019662.352671:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019662.352672:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019662.352674:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3928 00000100:00080000:2.0:1734019662.352676:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019662.352678:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019662.352684:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019662.352690:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334678272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019662.352695:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019662.352696:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019662.352697:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019662.352699:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3928 00000100:00080000:2.0:1734019662.352700:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019662.352703:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019662.352705:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019662.352710:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334678400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019662.352715:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019662.352716:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019662.352717:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019662.352719:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3928 00000100:00080000:2.0:1734019662.352720:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019662.352722:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019662.352724:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019662.352729:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334678528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019662.352734:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019662.352735:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019662.352736:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019662.352738:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3928 00000100:00100000:1.0:1734019662.352739:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334678144:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019662.352739:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019662.352741:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019662.352743:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019662.352747:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334678656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019662.352751:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019662.352752:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019662.352753:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019662.352755:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3928 00000100:00080000:2.0:1734019662.352756:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019662.352758:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019662.352760:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019662.352764:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334678784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019662.352768:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019662.352770:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019662.352771:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019662.352773:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3928 00000100:00080000:2.0:1734019662.352775:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019662.352777:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019662.352779:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019662.352782:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019662.352783:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334678912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019662.352798:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019662/real 1734019662] req@ffff8800a78a1880 x1818247334678144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019662.352809:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334678144/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019662.352814:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019662.352816:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019662.352819:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334678144:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019662.352822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334678272:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019662.352832:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019662.352836:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019662/real 1734019662] req@ffff8800a78a2680 x1818247334678272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019662.352842:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334678272/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019662.352846:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019662.352848:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019662.352850:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334678272:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019662.352852:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334678400:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019662.352860:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019662.352863:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019662/real 1734019662] req@ffff8800a78a1f80 x1818247334678400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019662.352868:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334678400/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019662.352872:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019662.352873:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019662.352875:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334678400:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019662.352889:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334678528:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019662.352896:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019662.352900:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019662/real 1734019662] req@ffff8800a78a3480 x1818247334678528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019662.352905:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334678528/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019662.352909:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019662.352910:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019662.352912:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334678528:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019662.352915:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334678656:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019662.352922:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019662.352925:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019662/real 1734019662] req@ffff8800a78a0700 x1818247334678656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019662.352930:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334678656/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019662.352934:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019662.352935:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019662.352936:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334678656:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019662.352939:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334678784:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019662.352945:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019662.352948:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019662/real 1734019662] req@ffff8800a78a3b80 x1818247334678784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019662.352953:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334678784/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019662.352957:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019662.352958:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019662.352960:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334678784:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019662.352962:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334678912:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019662.352969:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019662.352972:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019662/real 1734019662] req@ffff8800a78a0a80 x1818247334678912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019662.352978:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334678912/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019662.352981:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019662.352982:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019662.352984:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334678912:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019662.664586:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019663.666611:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019664.668642:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019665.670641:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019666.672674:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019667.360590:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019667.360597:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019667.360599:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019667.360605:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3933 00000100:00080000:2.0:1734019667.360611:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019667.360615:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019667.360639:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334679040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019667.360659:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019667.360661:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019667.360662:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019667.360664:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3933 00000100:00080000:2.0:1734019667.360666:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019667.360667:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019667.360673:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019667.360678:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334679168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019667.360683:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019667.360684:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019667.360685:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019667.360687:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3933 00000100:00080000:2.0:1734019667.360688:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019667.360690:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019667.360692:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019667.360696:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334679296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019667.360700:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334679040:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019667.360700:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019667.360702:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019667.360702:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019667.360704:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3933 00000100:00080000:2.0:1734019667.360705:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019667.360707:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019667.360708:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019667.360712:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334679424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019667.360716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019667.360718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019667.360718:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019667.360720:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3933 00000100:00080000:2.0:1734019667.360721:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019667.360723:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019667.360725:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019667.360728:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334679552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019667.360732:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019667.360733:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019667.360734:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019667.360735:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3933 00000100:00080000:2.0:1734019667.360737:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019667.360739:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019667.360740:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019667.360744:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334679680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019667.360747:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019667.360747:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019667.360749:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019667.360749:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019667.360751:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3933 00000100:00080000:2.0:1734019667.360752:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019667.360754:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019667.360755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019667.360759:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334679808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019667.360762:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019667/real 1734019667] req@ffff8800a78a0a80 x1818247334679040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019667.360774:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334679040/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019667.360781:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019667.360784:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019667.360787:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334679040:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019667.360792:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334679168:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019667.360805:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019667.360810:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019667/real 1734019667] req@ffff8800a78a3b80 x1818247334679168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019667.360818:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334679168/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019667.360823:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019667.360825:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019667.360827:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334679168:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019667.360844:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334679296:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019667.360855:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019667.360860:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019667/real 1734019667] req@ffff8800a78a0700 x1818247334679296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019667.360866:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334679296/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019667.360871:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019667.360873:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019667.360875:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334679296:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019667.360878:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334679424:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019667.360888:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019667.360892:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019667/real 1734019667] req@ffff8800a78a3480 x1818247334679424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019667.360899:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334679424/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019667.360904:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019667.360905:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019667.360908:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334679424:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019667.360912:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334679552:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019667.360922:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019667.360926:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019667/real 1734019667] req@ffff8800a78a1f80 x1818247334679552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019667.360934:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334679552/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019667.360939:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019667.360940:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019667.360942:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334679552:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019667.360946:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334679680:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019667.360955:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019667.360959:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019667/real 1734019667] req@ffff8800a78a2680 x1818247334679680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019667.360965:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334679680/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019667.360971:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019667.360972:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019667.360974:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334679680:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019667.360978:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334679808:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019667.360987:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019667.360992:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019667/real 1734019667] req@ffff8800a78a1880 x1818247334679808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019667.360999:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334679808/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019667.361004:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019667.361006:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019667.361008:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334679808:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019667.674629:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019668.676644:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019669.678637:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019670.680687:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019671.682662:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019672.368524:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019672.368531:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019672.368534:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019672.368540:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3939 00000100:00080000:2.0:1734019672.368547:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019672.368552:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019672.368577:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334679936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019672.368597:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019672.368600:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019672.368601:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019672.368605:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3939 00000100:00080000:2.0:1734019672.368607:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019672.368610:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019672.368617:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019672.368626:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334680064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019672.368634:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019672.368636:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019672.368637:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019672.368639:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334679936:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019672.368640:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3939 00000100:00080000:2.0:1734019672.368643:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019672.368646:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019672.368649:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019672.368656:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334680192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019672.368662:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019672.368664:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019672.368666:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019672.368668:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3939 00000100:00080000:2.0:1734019672.368670:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019672.368672:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019672.368675:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019672.368681:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334680320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019672.368684:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019672.368688:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019672.368690:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019672.368692:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019672.368694:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3939 00000100:00080000:2.0:1734019672.368696:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019672.368698:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019672/real 1734019672] req@ffff8800a78a1880 x1818247334679936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019672.368699:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019672.368702:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019672.368709:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334680448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019672.368711:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334679936/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019672.368716:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019672.368718:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1734019672.368719:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019672.368720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734019672.368721:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019672.368723:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3939 00000100:00100000:1.0:1734019672.368725:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334679936:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019672.368725:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019672.368727:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019672.368730:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019672.368736:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334680576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019672.368741:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334680064:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019672.368743:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019672.368746:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019672.368747:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019672.368750:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3939 00000100:00080000:2.0:1734019672.368752:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019672.368755:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019672.368755:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019672.368758:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019672.368760:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019672/real 1734019672] req@ffff8800a78a2680 x1818247334680064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019672.368765:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334680704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019672.368769:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334680064/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019672.368776:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019672.368778:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019672.368781:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334680064:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019672.368785:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334680192:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019672.368799:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019672.368805:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019672/real 1734019672] req@ffff8800a78a1f80 x1818247334680192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019672.368815:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334680192/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019672.368819:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019672.368820:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019672.368822:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334680192:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019672.368826:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334680320:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019672.368835:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019672.368838:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019672/real 1734019672] req@ffff8800a78a3480 x1818247334680320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019672.368844:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334680320/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019672.368848:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019672.368849:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019672.368850:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334680320:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019672.368854:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334680448:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019672.368861:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019672.368864:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019672/real 1734019672] req@ffff8800a78a0700 x1818247334680448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019672.368869:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334680448/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019672.368873:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019672.368874:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019672.368876:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334680448:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019672.368885:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334680576:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019672.368892:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019672.368895:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019672/real 1734019672] req@ffff8800a78a3b80 x1818247334680576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019672.368900:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334680576/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019672.368903:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019672.368905:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019672.368906:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334680576:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019672.368909:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334680704:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019672.368915:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019672.368918:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019672/real 1734019672] req@ffff8800a78a0a80 x1818247334680704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019672.368924:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334680704/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019672.368928:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019672.368929:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019672.368931:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334680704:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019672.684632:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019673.686785:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019674.688762:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019675.690676:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019676.692660:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019677.376554:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019677.376562:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019677.376565:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019677.376572:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3944 00000100:00080000:2.0:1734019677.376578:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019677.376583:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019677.376611:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334680832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019677.376636:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019677.376638:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019677.376639:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019677.376642:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3944 00000100:00080000:2.0:1734019677.376643:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019677.376645:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019677.376651:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019677.376657:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334680960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019677.376662:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019677.376664:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019677.376664:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019677.376667:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3944 00000100:00080000:2.0:1734019677.376668:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019677.376670:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019677.376673:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019677.376677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334681088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019677.376682:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019677.376683:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019677.376684:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019677.376686:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3944 00000100:00080000:2.0:1734019677.376687:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019677.376689:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019677.376692:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019677.376696:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334681216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019677.376700:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019677.376702:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019677.376703:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019677.376704:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3944 00000100:00080000:2.0:1734019677.376705:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019677.376707:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019677.376709:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019677.376710:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334680832:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019677.376713:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334681344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019677.376717:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019677.376719:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019677.376720:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019677.376721:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3944 00000100:00080000:2.0:1734019677.376723:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019677.376724:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019677.376727:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019677.376731:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334681472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019677.376735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019677.376737:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019677.376737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019677.376739:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3944 00000100:00080000:2.0:1734019677.376741:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019677.376742:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019677.376744:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019677.376748:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334681600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019677.376754:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019677.376768:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019677/real 1734019677] req@ffff8800a78a0a80 x1818247334680832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019677.376778:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334680832/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019677.376784:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019677.376786:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019677.376788:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334680832:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019677.376792:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334680960:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019677.376801:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019677.376804:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019677/real 1734019677] req@ffff8800a78a3b80 x1818247334680960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019677.376810:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334680960/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019677.376815:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019677.376816:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019677.376818:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334680960:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019677.376821:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334681088:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019677.376828:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019677.376832:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019677/real 1734019677] req@ffff8800a78a0700 x1818247334681088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019677.376837:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334681088/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019677.376840:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019677.376842:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019677.376843:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334681088:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019677.376846:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334681216:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019677.376853:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019677.376856:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019677/real 1734019677] req@ffff8800a78a3480 x1818247334681216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019677.376861:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334681216/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019677.376865:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019677.376866:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019677.376868:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334681216:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019677.376883:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334681344:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019677.376890:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019677.376893:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019677/real 1734019677] req@ffff8800a78a1f80 x1818247334681344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019677.376899:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334681344/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019677.376903:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019677.376904:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019677.376905:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334681344:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019677.376908:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334681472:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019677.376914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019677.376917:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019677/real 1734019677] req@ffff8800a78a2680 x1818247334681472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019677.376922:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334681472/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019677.376926:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019677.376927:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019677.376928:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334681472:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019677.376931:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334681600:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019677.376937:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019677.376940:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019677/real 1734019677] req@ffff8800a78a1880 x1818247334681600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019677.376946:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334681600/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019677.376950:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019677.376951:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019677.376952:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334681600:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019677.694679:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019678.696633:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019679.698773:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019680.700692:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019681.702724:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019682.384598:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019682.384607:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019682.384612:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019682.384622:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3949 00000100:00080000:2.0:1734019682.384631:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019682.384639:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019682.384676:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334681728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019682.384710:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019682.384713:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019682.384716:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019682.384720:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3949 00000100:00080000:2.0:1734019682.384723:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019682.384727:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019682.384736:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019682.384748:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334681856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019682.384758:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019682.384761:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019682.384763:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019682.384767:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3949 00000100:00080000:2.0:1734019682.384769:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019682.384773:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019682.384777:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019682.384786:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334681984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019682.384794:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019682.384798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019682.384799:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019682.384802:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3949 00000100:00080000:2.0:1734019682.384805:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019682.384808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019682.384812:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019682.384820:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334682112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019682.384822:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334681728:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019682.384829:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019682.384832:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019682.384834:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019682.384838:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3949 00000100:00080000:2.0:1734019682.384841:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019682.384844:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019682.384848:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019682.384855:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334682240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019682.384863:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019682.384866:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019682.384868:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019682.384871:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3949 00000100:00080000:2.0:1734019682.384882:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019682.384886:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019682.384890:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019682.384898:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334682368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019682.384906:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019682.384909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019682.384910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019682.384914:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334681856:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019682.384914:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3949 00000100:00080000:2.0:1734019682.384916:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019682.384920:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019682.384924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019682.384931:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334682496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019682.384935:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334681984:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019682.384958:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334682112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019682.385006:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334682240:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019682.385025:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334682368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019682.385039:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334682496:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019682.704708:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019683.706713:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019684.708679:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019685.390715:0:1971:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.116:988 00000400:00000100:3.0:1734019685.390756:0:1971:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.116@tcp at host 192.168.201.116 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019685.390772:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019685.390800:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019685.390807:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019685.390813:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019685.390818:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019685.390824:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000800:00000100:3.0:1734019685.390830:0:1971:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.116@tcp 00000100:00100000:1.0:1734019685.390909:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019682/real 1734019685] req@ffff8800a78a3480 x1818247334682112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019685.390982:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334682112/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019685.391002:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019685.391005:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019685.391010:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334682112:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019685.391018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019682/real 1734019685] req@ffff8800a78a0700 x1818247334682240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019685.391030:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334682240/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019685.391037:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019685.391040:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019685.391043:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334682240:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019685.391049:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019682/real 1734019685] req@ffff8800a78a3b80 x1818247334682368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019685.391059:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334682368/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019685.391066:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019685.391069:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019685.391074:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334682368:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019685.391084:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019682/real 1734019685] req@ffff8800a78a0a80 x1818247334682496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019685.391100:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334682496/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019685.391120:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019685.391124:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019685.391129:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334682496:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019685.391139:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019682/real 1734019685] req@ffff8800a78a1880 x1818247334681728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019685.391156:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334681728/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019685.391168:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019685.391172:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019685.391178:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334681728:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019685.391187:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019682/real 1734019685] req@ffff8800a78a2680 x1818247334681856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019685.391206:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334681856/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019685.391218:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019685.391222:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019685.391228:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334681856:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019685.391237:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019682/real 1734019685] req@ffff8800a78a1f80 x1818247334681984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019685.391255:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334681984/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019685.391276:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019685.391279:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019685.391285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334681984:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019685.710712:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019686.712705:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019687.392580:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019687.392587:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019687.392590:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019687.392597:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3954 00000100:00080000:2.0:1734019687.392604:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019687.392608:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019687.392635:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334682624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019687.392656:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019687.392658:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019687.392659:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019687.392661:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3954 00000100:00080000:2.0:1734019687.392663:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019687.392665:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019687.392671:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019687.392677:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334682752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019687.392682:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019687.392684:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019687.392685:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019687.392688:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3954 00000100:00080000:2.0:1734019687.392689:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019687.392691:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019687.392694:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019687.392699:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334682880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019687.392704:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019687.392705:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019687.392706:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019687.392708:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3954 00000100:00080000:2.0:1734019687.392709:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019687.392711:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019687.392713:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019687.392717:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334683008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019687.392721:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019687.392722:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019687.392723:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019687.392725:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3954 00000100:00080000:2.0:1734019687.392726:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019687.392728:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019687.392730:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334682624:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019687.392730:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019687.392734:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334683136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019687.392739:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019687.392740:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019687.392741:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019687.392743:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3954 00000100:00080000:2.0:1734019687.392745:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019687.392746:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019687.392749:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019687.392752:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334683264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019687.392756:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019687.392758:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019687.392759:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019687.392760:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3954 00000100:00080000:2.0:1734019687.392762:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019687.392763:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019687.392765:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019687.392769:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334683392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019687.392774:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019687.392788:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019687/real 1734019687] req@ffff8800a78a1f80 x1818247334682624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019687.392798:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334682624/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019687.392804:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019687.392806:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019687.392808:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334682624:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019687.392812:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334682752:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019687.392821:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019687.392825:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019687/real 1734019687] req@ffff8800a78a2680 x1818247334682752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019687.392830:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334682752/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019687.392834:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019687.392836:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019687.392838:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334682752:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019687.392841:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334682880:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019687.392849:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019687.392852:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019687/real 1734019687] req@ffff8800a78a1880 x1818247334682880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019687.392857:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334682880/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019687.392861:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019687.392862:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019687.392864:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334682880:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019687.392867:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334683008:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019687.392873:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019687.392876:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019687/real 1734019687] req@ffff8800a78a0a80 x1818247334683008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019687.392881:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334683008/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019687.392916:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019687.392918:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019687.392920:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334683008:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019687.392936:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334683136:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019687.392945:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019687.392950:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019687/real 1734019687] req@ffff8800a78a3b80 x1818247334683136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019687.392955:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334683136/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019687.392959:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019687.392960:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019687.392962:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334683136:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019687.392965:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334683264:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019687.392971:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019687.392974:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019687/real 1734019687] req@ffff8800a78a0700 x1818247334683264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019687.392979:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334683264/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019687.392982:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019687.392983:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019687.392985:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334683264:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019687.392988:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334683392:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019687.392994:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019687.392998:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019687/real 1734019687] req@ffff8800a78a3480 x1818247334683392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019687.393003:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334683392/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019687.393011:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019687.393012:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019687.393014:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334683392:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019687.714612:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019688.716615:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019689.718660:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019690.720592:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019691.722594:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019692.400611:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019692.400622:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019692.400627:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019692.400638:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3959 00000100:00080000:2.0:1734019692.400647:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019692.400655:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019692.400694:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334683520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019692.400735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019692.400739:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019692.400741:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019692.400745:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3959 00000100:00080000:2.0:1734019692.400749:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019692.400752:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019692.400761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019692.400773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334683648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019692.400782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019692.400785:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019692.400787:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019692.400791:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3959 00000100:00080000:2.0:1734019692.400793:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019692.400797:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019692.400825:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019692.400836:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334683776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019692.400845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019692.400848:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019692.400850:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019692.400854:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3959 00000100:00080000:2.0:1734019692.400857:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019692.400860:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019692.400880:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019692.400885:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334683904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019692.400888:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334683520:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019692.400890:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019692.400891:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019692.400892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019692.400894:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3959 00000100:00080000:2.0:1734019692.400895:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019692.400897:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019692.400900:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019692.400904:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334684032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019692.400908:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019692.400909:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019692.400910:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019692.400912:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3959 00000100:00080000:2.0:1734019692.400913:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019692.400915:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019692.400917:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019692.400921:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334684160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019692.400925:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019692.400927:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019692.400928:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019692.400930:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3959 00000100:00080000:2.0:1734019692.400931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019692.400932:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019692.400933:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019692.400935:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019692.400939:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334684288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019692.400945:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019692/real 1734019692] req@ffff8800a78a3480 x1818247334683520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019692.400956:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334683520/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019692.400963:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019692.400964:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019692.400967:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334683520:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019692.400971:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334683648:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019692.400980:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019692.400984:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019692/real 1734019692] req@ffff8800a78a0700 x1818247334683648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019692.400989:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334683648/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019692.400993:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019692.400995:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019692.400997:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334683648:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019692.401000:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334683776:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019692.401008:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019692.401011:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019692/real 1734019692] req@ffff8800a78a3b80 x1818247334683776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019692.401016:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334683776/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019692.401020:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019692.401022:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019692.401023:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334683776:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019692.401037:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334683904:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019692.401044:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019692.401047:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019692/real 1734019692] req@ffff8800a78a0a80 x1818247334683904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019692.401052:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334683904/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019692.401056:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019692.401058:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019692.401059:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334683904:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019692.401062:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334684032:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019692.401068:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019692.401072:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019692/real 1734019692] req@ffff8800a78a1880 x1818247334684032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019692.401077:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334684032/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019692.401081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019692.401082:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019692.401083:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334684032:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019692.401086:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334684160:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019692.401092:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019692.401095:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019692/real 1734019692] req@ffff8800a78a2680 x1818247334684160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019692.401100:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334684160/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019692.401103:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019692.401104:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019692.401106:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334684160:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019692.401109:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334684288:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019692.401115:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019692.401118:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019692/real 1734019692] req@ffff8800a78a1f80 x1818247334684288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019692.401123:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334684288/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019692.401130:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019692.401132:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019692.401133:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334684288:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019692.724712:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019693.726721:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019694.728610:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019695.730643:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019696.732592:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019697.408578:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019697.408585:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019697.408588:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019697.408596:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3964 00000100:00080000:2.0:1734019697.408603:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019697.408608:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019697.408635:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334684416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019697.408657:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019697.408660:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019697.408661:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019697.408663:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3964 00000100:00080000:2.0:1734019697.408665:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019697.408667:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019697.408673:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019697.408679:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334684544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019697.408685:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019697.408687:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019697.408688:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019697.408690:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3964 00000100:00080000:2.0:1734019697.408691:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019697.408693:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019697.408695:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019697.408700:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334684672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019697.408704:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019697.408706:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019697.408707:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019697.408708:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3964 00000100:00080000:2.0:1734019697.408710:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019697.408711:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019697.408713:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019697.408730:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334684800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019697.408735:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019697.408736:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019697.408737:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019697.408739:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3964 00000100:00080000:2.0:1734019697.408741:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019697.408742:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019697.408745:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019697.408749:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334684928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019697.408750:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334684416:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019697.408754:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019697.408755:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019697.408756:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019697.408758:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3964 00000100:00080000:2.0:1734019697.408760:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019697.408761:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019697.408763:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019697.408768:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334685056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019697.408779:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019697.408781:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019697.408781:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019697.408783:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3964 00000100:00080000:2.0:1734019697.408784:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019697.408786:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019697.408788:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019697.408792:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334685184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019697.408805:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019697.408819:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019697/real 1734019697] req@ffff8800a78a1f80 x1818247334684416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019697.408829:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334684416/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019697.408835:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019697.408837:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019697.408840:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334684416:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019697.408844:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334684544:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019697.408862:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019697.408866:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019697/real 1734019697] req@ffff8800a78a2680 x1818247334684544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019697.408871:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334684544/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019697.408875:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019697.408877:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019697.408879:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334684544:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019697.408882:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334684672:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019697.408890:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019697.408893:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019697/real 1734019697] req@ffff8800a78a1880 x1818247334684672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019697.408898:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334684672/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019697.408902:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019697.408903:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019697.408905:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334684672:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019697.408908:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334684800:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019697.408914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019697.408917:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019697/real 1734019697] req@ffff8800a78a0a80 x1818247334684800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019697.408922:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334684800/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019697.408926:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019697.408927:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019697.408929:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334684800:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019697.408944:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334684928:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019697.408953:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019697.408956:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019697/real 1734019697] req@ffff8800a78a3b80 x1818247334684928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019697.408962:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334684928/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019697.408966:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019697.408967:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019697.408968:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334684928:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019697.408971:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334685056:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019697.408977:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019697.408980:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019697/real 1734019697] req@ffff8800a78a0700 x1818247334685056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019697.408985:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334685056/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019697.408989:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019697.408990:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019697.408991:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334685056:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019697.408994:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334685184:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019697.409000:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019697.409004:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019697/real 1734019697] req@ffff8800a78a3480 x1818247334685184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019697.409012:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334685184/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019697.409016:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019697.409017:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019697.409019:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334685184:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019697.734801:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019698.736758:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019699.738760:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019700.740724:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019701.742759:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019702.416590:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019702.416600:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019702.416604:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019702.416614:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3969 00000100:00080000:2.0:1734019702.416623:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019702.416631:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019702.416700:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334685312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019702.416733:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019702.416737:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019702.416739:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019702.416744:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3969 00000100:00080000:2.0:1734019702.416748:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019702.416752:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019702.416761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019702.416772:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334685440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019702.416782:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019702.416785:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019702.416786:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019702.416790:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3969 00000100:00080000:2.0:1734019702.416793:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019702.416797:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019702.416802:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019702.416811:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334685568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019702.416820:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019702.416823:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019702.416824:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019702.416828:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3969 00000100:00080000:2.0:1734019702.416830:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019702.416833:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019702.416837:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019702.416845:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334685696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019702.416853:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019702.416857:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019702.416858:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019702.416862:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334685312:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019702.416862:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3969 00000100:00080000:2.0:1734019702.416865:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019702.416868:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019702.416872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019702.416879:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334685824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019702.416887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019702.416890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019702.416892:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019702.416895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3969 00000100:00080000:2.0:1734019702.416898:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019702.416901:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019702.416905:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019702.416913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334685952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019702.416920:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019702.416923:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019702.416925:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734019702.416927:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019702.416928:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3969 00000100:00080000:2.0:1734019702.416931:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019702.416934:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019702.416938:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019702.416945:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334686080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019702.416949:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019702/real 1734019702] req@ffff8800a78a3480 x1818247334685312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019702.416968:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334685312/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019702.416979:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019702.416983:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019702.416987:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334685312:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019702.416995:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334685440:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019702.417011:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019702.417018:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019702/real 1734019702] req@ffff8800a78a0700 x1818247334685440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019702.417029:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334685440/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019702.417038:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019702.417041:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019702.417044:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334685440:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019702.417050:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334685568:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019702.417063:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019702.417070:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019702/real 1734019702] req@ffff8800a78a3b80 x1818247334685568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019702.417080:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334685568/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019702.417087:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019702.417089:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019702.417092:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334685568:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019702.417117:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334685696:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019702.417130:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019702.417136:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019702/real 1734019702] req@ffff8800a78a0a80 x1818247334685696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019702.417147:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334685696/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019702.417154:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019702.417156:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019702.417159:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334685696:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019702.417166:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334685824:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019702.417179:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019702.417185:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019702/real 1734019702] req@ffff8800a78a1880 x1818247334685824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019702.417195:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334685824/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019702.417202:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019702.417204:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019702.417207:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334685824:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019702.417213:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334685952:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019702.417224:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019702.417230:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019702/real 1734019702] req@ffff8800a78a2680 x1818247334685952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019702.417240:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334685952/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019702.417248:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019702.417250:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019702.417253:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334685952:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019702.417258:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334686080:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019702.417271:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019702.417276:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019702/real 1734019702] req@ffff8800a78a1f80 x1818247334686080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019702.417287:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334686080/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019702.417293:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019702.417295:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019702.417298:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334686080:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019702.744723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019703.746710:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019704.748746:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019705.750745:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019706.752733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019707.424627:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019707.424636:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019707.424639:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019707.424648:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3974 00000100:00080000:2.0:1734019707.424655:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019707.424661:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019707.424688:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334686208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019707.424727:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019707.424732:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019707.424735:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019707.424741:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3974 00000100:00080000:2.0:1734019707.424745:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019707.424751:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019707.424761:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019707.424778:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334686336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019707.424791:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334686208:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019707.424793:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019707.424798:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019707.424800:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019707.424805:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3974 00000100:00080000:2.0:1734019707.424809:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019707.424815:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019707.424821:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019707.424833:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334686464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019707.424845:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019707.424845:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019707.424850:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019707.424852:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019707.424856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3974 00000100:00080000:2.0:1734019707.424860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019707.424864:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019707/real 1734019707] req@ffff8800a78a1f80 x1818247334686208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019707.424866:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019707.424872:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019707.424884:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334686592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019707.424887:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334686208/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019707.424897:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734019707.424901:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019707.424902:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019707.424904:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734019707.424905:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019707.424909:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3974 00000100:00100000:1.0:1734019707.424911:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334686208:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019707.424913:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019707.424918:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019707.424924:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019707.424935:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334686336:192.168.201.116@tcp:38:kworker.0 00000100:00100000:2.0:1734019707.424935:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334686720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019707.424949:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019707.424954:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019707.424957:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734019707.424960:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019707.424962:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3974 00000100:00080000:2.0:1734019707.424966:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019707.424971:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019707/real 1734019707] req@ffff8800a78a2680 x1818247334686336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019707.424971:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019707.424977:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019707.424989:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334686336/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019707.424989:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334686848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019707.425003:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019707.425003:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734019707.425006:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019707.425008:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019707.425011:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019707.425012:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334686336:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019707.425016:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3974 00000100:00080000:2.0:1734019707.425020:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019707.425021:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334686464:192.168.201.116@tcp:8:kworker.0 00000100:00080000:2.0:1734019707.425025:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019707.425030:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019707.425042:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:2.0:1734019707.425043:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334686976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019707.425053:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019707/real 1734019707] req@ffff8800a78a1880 x1818247334686464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019707.425069:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334686464/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019707.425081:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019707.425084:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019707.425089:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334686464:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019707.425098:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334686592:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019707.425124:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019707.425136:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019707/real 1734019707] req@ffff8800a78a0a80 x1818247334686592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019707.425149:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334686592/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019707.425156:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019707.425159:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019707.425162:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334686592:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019707.425179:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334686720:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019707.425196:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019707.425203:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019707/real 1734019707] req@ffff8800a78a3b80 x1818247334686720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019707.425213:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334686720/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019707.425221:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019707.425223:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019707.425226:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334686720:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019707.425231:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334686848:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019707.425243:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019707.425249:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019707/real 1734019707] req@ffff8800a78a0700 x1818247334686848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019707.425259:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334686848/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019707.425266:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019707.425268:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019707.425271:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334686848:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019707.425276:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334686976:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019707.425289:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019707.425295:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019707/real 1734019707] req@ffff8800a78a3480 x1818247334686976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019707.425305:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334686976/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019707.425312:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019707.425315:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019707.425318:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334686976:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019707.754754:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019708.756754:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019709.758721:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019710.760707:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019711.762752:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019712.432655:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019712.432669:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019712.432675:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019712.432689:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3979 00000100:00080000:2.0:1734019712.432702:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019712.432712:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019712.432757:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334687104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019712.432796:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019712.432800:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019712.432802:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019712.432807:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3979 00000100:00080000:2.0:1734019712.432811:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019712.432815:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019712.432826:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019712.432838:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334687232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019712.432849:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019712.432852:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019712.432854:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019712.432857:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3979 00000100:00080000:2.0:1734019712.432860:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019712.432863:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019712.432868:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019712.432877:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334687360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019712.432887:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019712.432890:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019712.432891:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019712.432895:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3979 00000100:00100000:1.0:1734019712.432896:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334687104:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019712.432897:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019712.432900:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019712.432904:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019712.432913:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334687488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019712.432921:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019712.432924:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019712.432925:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019712.432929:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3979 00000100:00080000:2.0:1734019712.432932:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019712.432935:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019712.432939:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019712.432946:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334687616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019712.432954:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019712.432957:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019712.432958:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019712.432962:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3979 00000800:00000100:1.0:1734019712.432965:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019712.432965:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019712.432968:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019712.432972:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019712.432979:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334687744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019712.432987:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019712/real 1734019712] req@ffff8800a78a3480 x1818247334687104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019712.432987:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019712.432990:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019712.432992:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019712.432995:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3979 00000100:00080000:2.0:1734019712.432997:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019712.433001:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019712.433005:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019712.433007:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334687104/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019712.433012:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334687872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019712.433018:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019712.433021:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019712.433027:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334687104:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019712.433034:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334687232:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019712.433051:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019712.433058:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019712/real 1734019712] req@ffff8800a78a0700 x1818247334687232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019712.433069:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334687232/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019712.433077:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019712.433080:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019712.433084:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334687232:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019712.433107:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334687360:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019712.433121:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019712.433127:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019712/real 1734019712] req@ffff8800a78a3b80 x1818247334687360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019712.433138:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334687360/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019712.433146:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019712.433148:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019712.433151:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334687360:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019712.433157:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334687488:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019712.433169:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019712.433174:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019712/real 1734019712] req@ffff8800a78a0a80 x1818247334687488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019712.433184:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334687488/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019712.433191:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019712.433193:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019712.433197:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334687488:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019712.433203:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334687616:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019712.433216:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019712.433222:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019712/real 1734019712] req@ffff8800a78a1880 x1818247334687616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019712.433233:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334687616/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019712.433240:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019712.433242:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019712.433245:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334687616:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019712.433251:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334687744:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019712.433262:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019712.433268:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019712/real 1734019712] req@ffff8800a78a2680 x1818247334687744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019712.433278:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334687744/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019712.433285:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019712.433287:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019712.433290:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334687744:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019712.433296:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334687872:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019712.433308:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019712.433314:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019712/real 1734019712] req@ffff8800a78a1f80 x1818247334687872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019712.433325:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334687872/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019712.433332:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019712.433334:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019712.433337:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334687872:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019712.764635:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019713.766749:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019714.768733:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019715.770708:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019716.772709:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019717.440605:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019717.440614:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019717.440618:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019717.440628:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3984 00000100:00080000:2.0:1734019717.440637:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019717.440644:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019717.440679:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334688000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019717.440714:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019717.440720:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019717.440723:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019717.440729:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3984 00000100:00080000:2.0:1734019717.440734:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019717.440740:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019717.440753:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019717.440773:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334688128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019717.440777:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334688000:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019717.440788:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019717.440792:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019717.440795:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019717.440800:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3984 00000100:00080000:2.0:1734019717.440804:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019717.440808:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019717.440815:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019717.440827:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334688256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019717.440839:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019717.440844:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734019717.440845:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019717.440846:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019717.440851:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3984 00000100:00080000:2.0:1734019717.440855:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019717.440859:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019717.440865:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019717.440868:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019717/real 1734019717] req@ffff8800a78a1f80 x1818247334688000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019717.440878:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334688384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019717.440891:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334688000/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019717.440891:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019717.440896:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019717.440898:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019717.440903:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3984 00000100:00080000:1.0:1734019717.440905:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019717.440908:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734019717.440909:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019717.440913:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000100:00100000:1.0:1734019717.440916:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334688000:192.168.201.116@tcp:250:kworker.0 00000008:00100000:2.0:1734019717.440918:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019717.440930:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334688512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019717.440942:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019717.440944:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334688128:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019717.440947:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019717.440950:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019717.440956:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3984 00000100:00080000:2.0:1734019717.440960:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019717.440965:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019717.440969:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019717.440972:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019717.440980:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019717/real 1734019717] req@ffff8800a78a2680 x1818247334688128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019717.440985:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334688640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019717.440997:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334688128/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019717.440999:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019717.441004:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019717.441007:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734019717.441011:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019717.441012:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3984 00000100:00080000:1.0:1734019717.441015:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00080000:2.0:1734019717.441016:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019717.441020:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334688128:192.168.201.116@tcp:38:kworker.0 00000100:00080000:2.0:1734019717.441021:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019717.441028:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019717.441029:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334688256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:2.0:1734019717.441039:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334688768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019717.441051:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019717.441060:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019717/real 1734019717] req@ffff8800a78a1880 x1818247334688256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019717.441078:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334688256/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019717.441090:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019717.441093:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019717.441099:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334688256:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019717.441108:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334688384:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019717.441133:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019717.441144:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019717/real 1734019717] req@ffff8800a78a0a80 x1818247334688384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019717.441158:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334688384/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019717.441167:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019717.441169:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019717.441173:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334688384:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019717.441191:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334688512:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019717.441209:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019717.441215:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019717/real 1734019717] req@ffff8800a78a3b80 x1818247334688512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019717.441226:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334688512/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019717.441234:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019717.441236:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019717.441239:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334688512:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019717.441244:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334688640:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019717.441256:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019717.441262:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019717/real 1734019717] req@ffff8800a78a0700 x1818247334688640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019717.441272:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334688640/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019717.441279:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019717.441282:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019717.441285:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334688640:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019717.441290:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334688768:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019717.441302:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019717.441308:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019717/real 1734019717] req@ffff8800a78a3480 x1818247334688768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019717.441318:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334688768/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019717.441326:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019717.441328:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019717.441331:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334688768:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019717.774742:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019718.776740:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019719.778709:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019720.780678:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019721.782695:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019722.448547:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019722.448555:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019722.448558:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019722.448567:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3989 00000100:00080000:2.0:1734019722.448574:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019722.448580:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019722.448609:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334688896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019722.448639:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019722.448642:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019722.448643:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019722.448646:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3989 00000100:00080000:2.0:1734019722.448649:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019722.448651:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019722.448659:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019722.448667:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334689024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019722.448673:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019722.448675:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019722.448676:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019722.448678:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3989 00000100:00080000:2.0:1734019722.448680:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019722.448683:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019722.448686:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019722.448692:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334689152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019722.448698:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019722.448700:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019722.448701:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019722.448703:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3989 00000100:00080000:2.0:1734019722.448705:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019722.448707:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019722.448710:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:2.0:1734019722.448716:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334689280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019722.448722:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334688896:192.168.201.116@tcp:250:kworker.0 00000100:00080000:2.0:1734019722.448722:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019722.448724:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019722.448725:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019722.448727:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3989 00000100:00080000:2.0:1734019722.448728:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019722.448731:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019722.448733:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019722.448738:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334689408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019722.448743:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019722.448745:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019722.448746:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019722.448748:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3989 00000100:00080000:2.0:1734019722.448750:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019722.448752:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019722.448755:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019722.448760:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334689536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019722.448765:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019722.448767:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019722.448768:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019722.448771:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3989 00000100:00080000:2.0:1734019722.448773:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019722.448774:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00080000:2.0:1734019722.448775:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019722.448778:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019722.448783:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334689664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019722.448790:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019722/real 1734019722] req@ffff8800a78a3480 x1818247334688896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019722.448803:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334688896/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019722.448811:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019722.448813:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019722.448817:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334688896:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019722.448821:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334689024:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019722.448833:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019722.448837:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019722/real 1734019722] req@ffff8800a78a0700 x1818247334689024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019722.448845:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334689024/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019722.448851:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019722.448853:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019722.448855:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334689024:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019722.448859:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334689152:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019722.448868:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019722.448872:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019722/real 1734019722] req@ffff8800a78a3b80 x1818247334689152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019722.448879:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334689152/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019722.448884:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019722.448885:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019722.448888:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334689152:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019722.448905:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334689280:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019722.448914:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019722.448919:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019722/real 1734019722] req@ffff8800a78a0a80 x1818247334689280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019722.448926:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334689280/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019722.448931:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019722.448932:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019722.448934:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334689280:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019722.448938:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334689408:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019722.448947:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019722.448951:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019722/real 1734019722] req@ffff8800a78a1880 x1818247334689408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019722.448957:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334689408/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019722.448962:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019722.448964:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019722.448966:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334689408:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019722.448969:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334689536:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019722.448977:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019722.448981:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019722/real 1734019722] req@ffff8800a78a2680 x1818247334689536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019722.448987:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334689536/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019722.448992:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019722.448993:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019722.448995:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334689536:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019722.448999:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334689664:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019722.449008:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019722.449012:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019722/real 1734019722] req@ffff8800a78a1f80 x1818247334689664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019722.449019:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334689664/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019722.449024:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019722.449026:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019722.449028:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334689664:192.168.201.116@tcp:8:kworker.0 00010000:00010000:1.0:1734019722.784805:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019723.786750:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019724.788723:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019725.790718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019726.792731:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019727.456585:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 99867e87-1bc2-4e49-bd45-a757ae2fdd76->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019727.456594:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019727.456598:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019727.456608:0:11157:0:(import.c:523:import_select_connection()) MGC192.168.201.116@tcp: connect to NID 192.168.201.116@tcp last attempt 3994 00000100:00080000:2.0:1734019727.456616:0:11157:0:(import.c:580:import_select_connection()) MGC192.168.201.116@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019727.456624:0:11157:0:(import.c:615:import_select_connection()) MGC192.168.201.116@tcp: import ffff8800b583b000 using connection MGC192.168.201.116@tcp_0/192.168.201.116@tcp 00000100:00100000:2.0:1734019727.456661:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1f80 x1818247334689792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019727.456695:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019727.456699:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019727.456701:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019727.456705:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3994 00000100:00080000:2.0:1734019727.456708:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019727.456712:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a571000: import ffff8800b6116800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019727.456721:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019727.456733:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a2680 x1818247334689920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019727.456743:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019727.456747:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019727.456748:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019727.456752:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3994 00000100:00080000:2.0:1734019727.456754:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019727.456758:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a571000: import ffff8800b6117800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019727.456762:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1734019727.456771:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a1880 x1818247334690048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019727.456780:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 8c598caf-7141-4c19-8566-0b7731516db9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019727.456783:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019727.456784:0:11157:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019727.456787:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: connect to NID 192.168.201.116@tcp last attempt 3994 00000100:00080000:2.0:1734019727.456790:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019727.456794:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a571000: import ffff8800aa959000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019727.456798:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 69632, lost: 0. 00000100:00100000:1.0:1734019727.456799:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334689792:192.168.201.116@tcp:250:kworker.0 00000100:00100000:2.0:1734019727.456806:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0a80 x1818247334690176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019727.456815:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019727.456818:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019727.456820:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019727.456823:0:11157:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3994 00000100:00080000:2.0:1734019727.456826:0:11157:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019727.456829:0:11157:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff88012a5ff800: import ffff88012e28f000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019727.456833:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019727.456841:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3b80 x1818247334690304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019727.456848:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019727.456851:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019727.456853:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019727.456856:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3994 00000100:00080000:2.0:1734019727.456859:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019727.456862:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff88012a5ff800: import ffff88012e28c800 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000800:00000100:1.0:1734019727.456864:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000008:00100000:2.0:1734019727.456866:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019727.456873:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a0700 x1818247334690432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019727.456881:0:11157:0:(pinger.c:254:ptlrpc_pinger_process_import()) 84392f61-a30b-4a4c-b6a5-f13f1ae7284e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019727.456884:0:11157:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019727.456885:0:11157:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019727.456886:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019727/real 1734019727] req@ffff8800a78a1f80 x1818247334689792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019727.456889:0:11157:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: connect to NID 192.168.201.116@tcp last attempt 3994 00000100:00080000:2.0:1734019727.456891:0:11157:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019727.456894:0:11157:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff88012a5ff800: import ffff88012e28b000 using connection 192.168.201.116@tcp/192.168.201.116@tcp 00000008:00100000:2.0:1734019727.456898:0:11157:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019727.456905:0:11157:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a78a3480 x1818247334690560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019727.456906:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1f80 x1818247334689792/t0(0) o250->MGC192.168.201.116@tcp@192.168.201.116@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019727.456917:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b583b000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019727.456920:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019727.456925:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99867e87-1bc2-4e49-bd45-a757ae2fdd76:1980:1818247334689792:192.168.201.116@tcp:250:kworker.0 00000100:00100000:1.0:1734019727.456933:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334689920:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019727.456949:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019727.456956:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019727/real 1734019727] req@ffff8800a78a2680 x1818247334689920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019727.456967:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a2680 x1818247334689920/t0(0) o38->lustre-MDT0000-mdc-ffff88012a571000@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019727.456976:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6116800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019727.456979:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019727.456983:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334689920:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019727.456989:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334690048:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019727.457002:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019727.457008:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019727/real 1734019727] req@ffff8800a78a1880 x1818247334690048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019727.457018:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a1880 x1818247334690048/t0(0) o8->lustre-OST0000-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019727.457026:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800b6117800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019727.457028:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019727.457031:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334690048:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019727.457056:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334690176:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019727.457069:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019727.457075:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019727/real 1734019727] req@ffff8800a78a0a80 x1818247334690176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019727.457086:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0a80 x1818247334690176/t0(0) o8->lustre-OST0001-osc-ffff88012a571000@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019727.457093:0:1980:0:(import.c:86:import_set_state_nolock()) ffff8800aa959000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019727.457095:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019727.457098:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8c598caf-7141-4c19-8566-0b7731516db9:1980:1818247334690176:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019727.457105:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334690304:192.168.201.116@tcp:38:kworker.0 00000800:00000100:1.0:1734019727.457118:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019727.457124:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019727/real 1734019727] req@ffff8800a78a3b80 x1818247334690304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019727.457134:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3b80 x1818247334690304/t0(0) o38->lustre-MDT0000-mdc-ffff88012a5ff800@192.168.201.116@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019727.457141:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28f000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019727.457143:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019727.457146:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334690304:192.168.201.116@tcp:38:kworker.0 00000100:00100000:1.0:1734019727.457151:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334690432:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019727.457163:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019727.457169:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019727/real 1734019727] req@ffff8800a78a0700 x1818247334690432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019727.457178:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a0700 x1818247334690432/t0(0) o8->lustre-OST0000-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019727.457186:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28c800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019727.457188:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019727.457191:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334690432:192.168.201.116@tcp:8:kworker.0 00000100:00100000:1.0:1734019727.457196:0:1980:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334690560:192.168.201.116@tcp:8:kworker.0 00000800:00000100:1.0:1734019727.457209:0:1980:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.116@tcp 00000100:00100000:1.0:1734019727.457258:0:1980:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019727/real 1734019727] req@ffff8800a78a3480 x1818247334690560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019727.457270:0:1980:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a78a3480 x1818247334690560/t0(0) o8->lustre-OST0001-osc-ffff88012a5ff800@192.168.201.116@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019727.457277:0:1980:0:(import.c:86:import_set_state_nolock()) ffff88012e28b000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019727.457280:0:1980:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.116@tcp failed (-110) 00000100:00100000:1.0:1734019727.457283:0:1980:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a78a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:84392f61-a30b-4a4c-b6a5-f13f1ae7284e:1980:1818247334690560:192.168.201.116@tcp:8:kworker.0 00010000:00010000:3.0:1734019727.794718:0:4750:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734019728.796695:0:10913:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 72680 lines, 72680 kept, 0 dropped, 0 bad.